Spark Summit 2013 brought the Apache Spark community together on December 2-3, 2013 at the Hotel Nikko in San Francisco. It featured production users of Spark, Shark, Spark Streaming and related projects.
Spark’s user friendly API enables developers with rapid development and high levels of productivity. Frameworks like Mesos and Yarn provide resource sharing but are designed around users acting in isolation. Tachyon offers support for in memory file-sharing among users, but applications are still written disjointly.
In this talk I will present Laburnum, the framework we use at Quantifind to facilitate sharing both data and code among multiple application developers. Such a framework allows us to present a unified workflow to our end users through a web interface, enabling ad-hoc queries over large data sets without having to learn Spark for themselves.
I’ll highlight the work we’ve put into making this shared memory and code model possible. I will also talk about running applications (spark workflows) both within and outside of this framework, hot-swapping new applications into an existing Laburnum instance, and our work on accessing a running instance through the REPL.