EclairJS = Node.Js + Apache Spark

Slides PDF Video

Digital businesses are increasingly looking toward sophisticated analytics to improve their customer interactions, and today’s interactive and user-facing applications are developed with technologies such as Node.js, Ruby on Rails, and PHP. Node.js in particular can handle very large numbers of simultaneous requests which contributes to its popularity, however it is not well-suited to the heavier workloads demanded by many analytics and so these workloads are typically handed off to back-end engines. Apache Spark is an ideal engine for this purpose because it is easily scalable, it provides a growing assortment of ML analytics, and within a single processing context it supports both static and streaming data, a SQL database, and a graph engine. Spark has no native API for Node.js or JavaScript and so the EclairJS project was started to provide developers with such an API. EclairJS enables developers to create Node.js applications and to call Spark functions in JavaScript, and to do so in a style that requires no special language constructions. Indeed, developers can write the lambda functions that are passed as arguments to Spark functions such as .map in JavaScript. In this presentation, I’ll show an EclairJS streaming and analytical application, and use the code to frame a technical description of EclairJS including coding style and the use of Spark operators and datastructures. In addition, I’ll describe the basic architectural components of EclairJS including the EclairJS-Node and EclairJS-Nashorn components and how they relate to Spark.

David Fallside,  at IBM

About David

David Fallside leads an emerging technology team at IBM that is building an Apache Spark API for Node.js developers. The team’s previous projects have focussed on creating LoB tools for Spark and Hadoop, and information engineering for IBM’s Watson. He was also responsible for creating the Apache Derby project, and has worked in several W3C groups including its Advisory Board. David has a PhD in Cognitive Psychology from Carnegie Mellon University.