Senior Data Engineer

The successful candidate will be working on cutting-edge technologies to perform statistical profiling, inference, classification, clustering and predictive analysis. As a key member of the Data Engineering team, you will be working on large datasets of Security Data and implement Data mining, Data parsing contributing to data availability building machine learning models to help derive new insights to defend against cyber-attacks.

• Interact and aid to provide data for other investigative teams with time sensitive, critical investigations.
• Work with cross functional teams to fully understand business requirements and desired business outcomes
• Assist in scoping and designing analytic data assets, implementing modeled attributes and contributing to brainstorming sessions
• Build and maintain a robust data engineering process to develop and implement self-serve data and tools
• Find opportunities to create, automate and scale repeatable analyses or build self-service tools for business users
• Execute data engineering projects ranging from small to large either individually or as part of a project team
• Experience on working with very large data sets and knowledge of building programs that leverage Massively Parallel Processing (MPP) Data warehouse platforms.
• The engineer will be working on Big Data technologies like Hadoop and tools and security related datasets doing Data parsing, Data mining activities and writing parsers.
• Responsible for loading data from several disparate datasets, documentation and debugging applications.

• 5 years of work experience with a Bachelor’s Degree or an Advanced Degree.
• 4+ years of software development experience (with a concentration in data centric initiatives), with demonstrated expertise in leveraging standard development best practice methodologies.
• Experience with Data Engineering with production pipelines, utilizing data engineering techniques that enable statistical modeling solutions to solve business problems.
• Experience with various Opensource Hadoop distribution systems and technology stack like Spark, oozie, airflow etc.

• Extensive experience with big data technologies (Hadoop, Hive, Druid, etc.) tools for large scale data processing, data transformation and machine learning pipelines.
• Ability to create and manage big data pipeline using Syslog-ng, kafka, flume, Airflow etc.
• Hands-on expertise with Java or Python, Spark and any scripting languages. Go lang and Scala is a plus.
• Experience with highly distributed, scalable, concurrent and low latency systems working with one or more of the following database technologies: MySQL, Postgres SQL, NoSQL DB, Elastic Search.
• Experience with data visualization and business intelligence tools like Power BI, Tableau, Microstrategy, or other programs.
• Experience in developing Rest API.
• Experience with Continuous Integration and Automated Test tools such as Jenkins, Artifactory, Git, Ansible.
• Experience working in an Agile and Test-Driven Development environment.
• Demonstrated intellectual and analytical rigor, strong attention to detail, team oriented, energetic, collaborative, diplomatic, and flexible style.