Vacancy expired!
Senior Data Engineer Location: Plano, TX/ HybridDuration: 12 months Description: Job Summary:Ability to design & build real-time applications using Apache Kafka & Spark Streaming.Develop and execute data pipeline testing processes and validate business rules and policies.Ability to design, build and unit test applications on Spark framework on Python.Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.Ability to solve complex data-driven scenarios and triage defects and production issues.Develop & review technical documentation for artifacts delivered.Ability to learn-unlearn-relearn concepts with an open and analytical mindset. Requirements:Minimum 7 years of experience in data engineeringMinimum 3+ years’ experience with Kafka Streams/building data intensive streaming applications (stream processing, e.g. Kafka, Spark Streaming)Minimum 3 years of extensive experience in designing, building, and deployment of PySpark-based applications.Minimum 2 year of experience with a public cloud (AWS, Microsoft Azure, or a large Tech company Cloud)Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.Excellent communication skills and Good Customer Centricity.Strong Target & High Solution Orientation. Call notes:Team Info:Financial Services - EMD, provide data to all LOB partnersBuild frameworks and utilities5 Bigdata frameworks, Requirement enhancements & optimizationSecured S3 buckets Role Info:Hands on Sr. Data Engineer with strong Python/Scala/JavaBuilding modern real time data pipelineswork with ambiguous solutions, evaluate different optionsLeverage open-source Apache framework Top Skills:7+ Yrs. exp is a mustSpark exp is a mustAWS any exp is a mustJava/Scala /Flink exp is requiredKafka/ kinesis exp is required
- ID: #44700555
-
State: Texas
Plano
75023
Plano
USA
- City: Plano
- Salary: $20 - $80
- Job type: Contract
- Showed: 2022-08-08
- Deadline: 2022-09-25
- Category: Et cetera