Sr. Data Engineer Lead

06 Nov 2024

Vacancy expired!

Combining consumer surveys, retail point-of-sale data, and cutting-edge analytics, The NPD Group delivers trusted retail industry research to the world’s most successful brands and retailers. We also offer careers that reward innovation and provide growth opportunities for forward-thinking problem-solvers who join our team. Speed your career growth, while working with the best and brightest in the market research industry.

Position OverviewHeadquartered in Port Washington, NY with offices around the globe, The NPD Group is a leader in the collection, processing, and delivery of point-of-sale and consumer tracking data in more than 20 industries. We are expanding and seeking a senior data engineering lead who has hands on experience working with cloud and Big data platforms. You will be responsible for data engineering, data analysis and user support primarily focused on Azure Databricks and Cloudera Hadoop. You will have strong hands-on technical skills as well as being able to mentor and train other Engineers with programming languages such as Python and Scala, including tuning complex big data spark applications.

Responsibilities
  • Design, develop, and maintain highly scalable cloud-based data processing applications
  • Manage and mentor a team of 3-5 data engineers
  • Write efficient, reusable and well documented code
  • Deliver highly complex big data projects using Databricks, Data Factory and other Azure services
  • Maintain and tune existing Hadoop/Hbase applications and migrate to Azure cloud
  • Actively participate in daily agile / scrum meetings

Qualifications:
  • 7+ years of software development and design
  • 2+ years of leading a team of 3-5 software developers
  • Experience in developing and tuning spark applications using Databricks
  • Excellent understanding of spark architecture, spark SQL, data frames and tuning big data spark applications
  • Excellent with Python, Scala, SQL and shell scripting
  • Hands-on experience with cloud platforms like Azure and AWS
  • Experience in Database Design and Data Modelling
  • Excellent problem solving and analytical skills
  • Strong diagramming skills – flowcharts, data flows, etc.
  • Solid written and verbal communication skills
  • Experience with Synapse, Snowflake or Google BigQuery is a plus
  • Bachelor's degree in Computer Science or related field

  • ID: #22290648
  • State: New York New york city 10001 New york city USA
  • City: New york city
  • Salary: Depends on Experience
  • Job type: Permanent
  • Showed: 2021-11-06
  • Deadline: 2021-12-27
  • Category: Et cetera