Vacancy expired!
We are seeking a Senior Data Pipeline Engineer for a consulting engagement with a global, household-name client. This position can be in either: Los Angeles, Seattle, or NYC. What you will do:
- Think and communicate critically about architecture, design, and best practices and guide your team to adopting them.
- Design data systems that allow managed growth of the data model to minimize risk and cost of change.
- Write transformation and validation code that applies complex data aggregation and calculation using SQL and Python.
- Drive implementation of automated testing for data pipelines within a CI environment
- 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow
- 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR
- Experience with binary data serialization formats such as Parquet
- Deep understanding of data structures and algorithms
- Understanding of code versioning tools such as GIT
- Have a passion for data solutions
- Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue
- Experience integrating with Ad Tech platforms such as Operative and STAQ
- Exposure and opinions regarding alternate orchestration tooling beyond Airflow
- Understanding of SOX compliance needs and how they affect system design.
- Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS
- Can flex between Python and JavaScript/Typescript.
- Aurora/Hive (databases)
- Spark (large-scale data processing)
- Airflow (workflow management)
- Docker (software packaging and delivery)
- AWS (development and hosting)
- ID: #42833388
- State: Washington Seattle-tacoma 98101 Seattle-tacoma USA
- City: Seattle-tacoma
- Salary: Depends on Experience
- Job type: Contract
- Showed: 2022-06-11
- Deadline: 2022-08-09
- Category: Et cetera