Vacancy expired!
Please, no third parties. Permanent residents only.This can be a fully remote position.Main Duties & Responsibilities:- Develop the delivery of company’s data platform supporting ingestion of over ten thousand datasets, stores over a petabyte of spatial, relational and raster data; support services serving over five billion API calls per month.- Work as part of a collaborative, Agile team to deliver solutions and services for data platform to support delivery of large-scale data lake.- Participate in architecture, design and development reviews for data platform services leveraging best available big data tools and technologies such as Presto, Spark, Alluxio, etc.- Engage with cross organizational teams collaborating on data ingestion services and data engineering to develop single consistent series of services and solutions.- Work with evolving requirements based on results of teams continuing investigation, development and customer feedback.- Adhere to best practices around source code versioning, automated testing, and dependency management.- Investigate and resolve technical and non-technical issues, resolving critical incidents in timely manner and with a through root cause analysis.- Contribute to company's overall technology strategy and roadmap as an active member of its architectural leadership team.Skills & Requirements:-7+ years of software engineering experience including 3+ years of experience with Big Data systems and cloud architecture.- Big Data architecture and systems, including distributed data processing systems (Spark and/or Dask preferred).- Distributed data storage systems (Parquet and/or HDFS preferred).- Low-latency data lake query architectures (Alluxio preferred) and real-time streaming systems (Kafka preferred).- Data lake design strategies for metadata, ontology, governance, authorization, etc.- Test automation for data quality, data flow, and API endpoints.- Data engineering techniques for Big Data, including data automation frameworks (such as Airflow or Prefect), metadata management (such as Amundsen) and process management strategies.- Infrastructure management and automation, such as Kubernetes, Terraform and Chef.- Cloud infrastructure management, ideally with experience in AWS, including both technical aspects, such as solutions architecture, and non-technical aspects, such as financial planning.- Modern practices around Agile development, release management, continuous integration, system reliability.- Execute on a data platform strategy in collaboration with team members, architects, product managers and other groups across the business.- Collaborate as significant individual technical contributor to meet overall team objectives and goals.- Stay up to date on emerging technologies, standards, and protocols.- Fundamentals of computer science and software engineering.- Bachelors Degree in Computer Science or related field is preferred. tags: senior data engineer, senior database engineer, senior big data engineer, senior software engineer, senior cloud engineer
- ID: #49289683
-
State: California
Irvine
92617
Irvine
USA
- City: Irvine
- Salary: Depends on Experience
- Job type: Permanent
- Showed: 2023-02-18
- Deadline: 2023-04-18
- Category: Et cetera