Andrew Or, Software Engineer at Databricks

Andrew Or

Software Engineer, Databricks

Andrew is a Spark PMC member. In the past, he has contributed several large features to the project, including event logging, external spilling, history server, dynamic allocation, and DAG visualization on the SparkUI. He is an active maintainer of the Spark on YARN integration component.


Deep Dive: Apache Spark Memory Management

Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk… Read more