SESSION

Dr. Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop

Slides PDF Video

Dr. Elephant helps improve Spark and Hadoop developer productivity and increase cluster efficiency by making clear recommendations on how to tune workloads and configurations. Originally developed by LinkedIn, Dr. Elephant is now in use at multiple sites.

This session will explore how Dr. Elephant works, the data it collects from Spark environments and the customizable heuristics that generate tuning recommendations. Learn how Dr. Elephant can be used to improve production cluster operations, help developers avoid common issues, and green light applications for use on production clusters.

Session hashtag: #SFdev18

Carl Steinbach, Senior Software Engineer at LinkedIn

About Carl

Carl Steinbach is a Senior Staff Software Engineer at LinkedIn where he leads the Grid Platform Team. In 2014 he launched the Dr. Elephant project with the goal of improving user productivity and the overall efficiency of LinkedIn’s cluster infrastructure.

Simon King, Software Engineer at Pepperdata

About Simon

Simon King is a software engineer at Pepperdata where he is the engineering lead on Pepperdata work with both the open source Dr. Elephant project and integration of Dr. Elephant technology with the Pepperdata Application Profiler. Prior to Pepperdata, Simon worked at Microsoft and Yahoo on a variety of technology projects.