Speaker

Yu Peng, Data Engineer at Databricks

Yu Peng

Data Engineer, Databricks

Yu Peng is a data engineer on the data science team at Databricks. He’s working on building Databricks’ real-time logging pipeline on top of Kinesis and Spark. Prior to joining Databricks, he was a Tech Lead of ETL and external Reporting team in Rocket Fuel. He received his Ph.D in Computer Science from The University of Hong Kong in 2013.

Sessions

Databricks' Data Pipelines: Journey And Lessons Learned

With components like Spark SQL, MLlib, and Streaming, Spark is a unified engine for building data applications. In this talk, we will take a look at how we use Spark on our own Databricks platform… Read more