Lessons Learned from Managing Thousands of Production Apache Spark Clusters Daily

Slides PDF Video

At Databricks, we have a unique view into hundreds different companies using Apache Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common manageability, debugging, and visibility issues that our users run into. This talk will first show some representatives of these common issues. Then, we will show you what we have done and have been working on in Databricks to make Spark clusters easier to manage, monitor, and debug.

Session hashtag: #SFexp19

Henry Davidge, Software Engineer at Databricks

About Henry

Henry Davidge is a software engineer at Databricks where he focuses on building the cluster management infrastructure. Before Databricks, he graduated from Yale University with a BS in computer science.