Software Engineer, Spark Team

About the role: Partners with stakeholders and leads team efforts to build and maintain backend services and solutions to support user-facing products, downstream services, or infrastructure tools and platforms used across Uber. Spark is our main batch big data analytics query engine at Uber. It accounts for the majority of our data compute resources, powers most of our major ingestion, ETL, data science, and machine learning workloads. Spark at Uber is used by over 2.5K weekly active users, who run 3M+ applications every week. As part of the Spark team, you will design, implement, optimize, and manage large scale distributed computing infrastructure that will impact multiple business use cases at Uber scale. You will also have the opportunity to collaborate with the open source community for Spark and related domains. About the Team: Uber has unique challenging data use cases at the intersection of massive data, realtime requirements, privacy and security considerations, geospatial and large distributed machine learning. The Apache Spark team powers these use-cases with a Spark as a service platform. This platform is the backbone of all large-scale data processing at Uber, and powers the core matching algorithms that powers our many business lines. Spark is also the data fabric for the company that connects many diverse data sources ranging across HDFS, Pinot, Kafka, Cassandra, ElasticSearch, etc. We have a diverse user base ranging from machine learning scientists to data analysts. Our workload ranges from very complex RDD processing frameworks for ingestion, that shuffle hundreds of petabytes a day, complex distributed deep learning using Horovod and Michaelangelo, to notebooks and plain old SQL ETL. Many of our jobs are themselves deep million line frameworks that further enable Uber's engineers to do experimentation, machine learning, log analytics etc. Minimum qualifications: BS equivalent in Computer Science, Engineering, Mathematics or related field OR 2-years full-time Software Engineering work experience, WHICH INCLUDES 1-years total technical software engineering experience in one or more of the following areas: Programming language (e.g. C, C, Java, Python, or Go) Note the 1-years total of specialized software engineering experience may have been gained through education and full-time work experience, additional training, coursework, research, or similar (OR some combination of these). The years of specialized experience are not necessarily in addition to the years of Education & full-time work experience indicated. Technical skills: Preferred: Distributed systems