Member of Technical Staff- Spark on YARN (Hortonworks)

Apache Hadoop YARN is the data operating system at the core of Hadoop, responsible for scheduling, resource management and workload management at Internet scale. Apache SPARK is a programming model and execution engine for large scale data processing. We are looking for candidates with experience in large-scale, distributed systems to help drive the implementation of SPARK on YARN. Your primary focus will be to ensure the best possible experience for multi-tenant mission critical enterprise deployments of Spark on YARN, with emphasis on scalability and reliability.


  • Experience with large-scale, distributed systems design and development with strong understanding of scaling, performance and scheduling.
  • Hands on programmer, strong in data structures and programming practices.
  • Java & Scala experience desirable.
  • Experience using MapReduce or other parallel programming techniques and experience using or developing AWS or other Cloud platforms.
  • Experience using multi-tenancy systems features such as Linux containers, cgroups.
  • Experience using projects in Apache Hadoop ecosystem such as Pig, Hive, HBase etc. is a big plus.
  • Experience contributing to the Apache Spark ecosystem is a big plus.
  • Strong oral and written communication skills
  • Experience contributing to Open Source projects is desirable.
  • Ability to work in an agile and collaborative setup within an engineering team.


As one of Nasdaq’s newest public companies, Hortonworks is experiencing extraordinary growth as we deliver essential support to the burgeoning big data community. Our 100% Open Source Hadoop Platform helps the Enterprise store, manage, process, and analyze large amounts of structured and unstructured data.

Hortonworks is the leader in accelerating business transformations with Open Enterprise Hadoop by developing, distributing and supporting an enterprise-scale data platform built entirely on open source technology including Apache™ Hadoop®. Our team comprises the largest contingent of builders and architects within the Hadoop ecosystem who represent and lead the broader enterprise requirements within these communities.

The Hortonworks Data Platform provides an open platform that deeply integrates with existing IT investments and upon which enterprises can build and deploy Hadoop-based applications.

Hortonworks has deep relationships with the key strategic data center partners that enable our customers to unlock the broadest opportunities from Hadoop.

For more information, visit

Hortonworks and HDP are registered trademarks or trademarks of Hortonworks, Inc. and it’s subsidiaries in the United States and other jurisdictions.





Job posted 10/28/2015