Spark Summit 2013 brought the Apache Spark community together on December 2-3, 2013 at the Hotel Nikko in San Francisco. It featured production users of Spark, Shark, Spark Streaming and related projects.
Virtualization of enterprise data centers has many benefits but also challenges to operations management, in particular the dynamism in how resource consumers and suppliers are connected. CloudPhysics is creating an operations management SaaS product to address such challenges. Our service has hundreds of active users. Each day more than 100 billion data samples are collected from over 100K virtual machines and physical servers. We are using Spark and Spark Streaming to build cross-user analysis tools. Such tools detect patterns and trends in the collective data set so that we can alert users of performance and configuration anomalies. We can also suggest best practice tunings using the aggregate analysis results. We will present the architecture of our tools, share our experiences running Spark and our thinking on future developments.