Senior Data Pipeline Engineer

11 Jun 2024

Vacancy expired!

We are seeking a Senior Data Pipeline Engineer for a consulting engagement with a global, household-name client. This position can be in either: Los Angeles, Seattle, or NYC. What you will do:

  • Think and communicate critically about architecture, design, and best practices and guide your team to adopting them.
  • Design data systems that allow managed growth of the data model to minimize risk and cost of change.
  • Write transformation and validation code that applies complex data aggregation and calculation using SQL and Python.
  • Drive implementation of automated testing for data pipelines within a CI environment
  • 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow
  • 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR
  • Experience with binary data serialization formats such as Parquet
  • Deep understanding of data structures and algorithms
  • Understanding of code versioning tools such as GIT
  • Have a passion for data solutions
Nice to Have:
  • Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue
  • Experience integrating with Ad Tech platforms such as Operative and STAQ
  • Exposure and opinions regarding alternate orchestration tooling beyond Airflow
  • Understanding of SOX compliance needs and how they affect system design.
  • Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS
  • Can flex between Python and JavaScript/Typescript.
Technical Environment
  • Aurora/Hive (databases)
  • Spark (large-scale data processing)
  • Airflow (workflow management)
  • Docker (software packaging and delivery)
  • AWS (development and hosting)

We do not work with 3rd party employers. Visa Sponsorship NOT available.

About Moorecroft SystemsMoorecroft Systems has been providing IT consulting services to clients nationwide for more than 2 decades. We only work directly with our own clients, which include a number of the most admired global and technology progressive corporations.Our values include honesty, integrity, and professionalism, we are ONLY interested in representing consultants that have the same values.

  • ID: #42833388
  • State: Washington Seattle-tacoma 98101 Seattle-tacoma USA
  • City: Seattle-tacoma
  • Salary: Depends on Experience
  • Job type: Contract
  • Showed: 2022-06-11
  • Deadline: 2022-08-09
  • Category: Et cetera