PySpark Developer

PySpark DeveloperMontvale NJ 07645Experience: 6+ years Job Summary:

• Provide end-to-end technical ownership and delivery for data supply chain which includes Enterprise Data Lake (on-prem & cloud) Data quality and Enterprise Datawarehouse.
  • Hands-on experience with building optimizing the data pipe
    • CI/CD integrated
      • build and deployment automation configuration management test automation solutions Deep hands-on
      • experience with Hortonworks Hadoop Hive HBase Spark Kafka Synapse/Snowflake Python R SQL Java Scala RStudio Spark RDDs and Data Frames Ambari Ranger Kerberos Atlas and Collibra etc.
    • Hands-on experience in architecting and implementing end to end Azure cloud big data solutions as well as with implementing Real-time Solutions & data integrations and Big Data Management Tool (Zaloni) is a plus.
  • Experience in large-scale data warehousing implementations and high-performance ETL tools is required.
    • Solid understanding of Azure Cloud Stack including ADF Data flows Event Hub Databricks HDInsight SQL Server Azure DevOps.
  • Professional training and certifications in various big data solutions (preferred).

• Developer bachelor’s in science engineering or equivalent.
• Understand requirements build codes support testing and fix defects guide developers in the course of development activities in order to develop high standard stable codes within the limits and clients’ processes standards and guidelines.
• Project Planning and Set-up- Understand the project scope identify activities/ tasks task-level estimates schedule dependencies risks and provide inputs to Module Lead for review Provide inputs to testing strategy configuration deployment hardware/software requirement etc.
• Review plan and provide feedback on gaps timeline and execution feasibility etc. as required in the project Participate in KT sessions conducted by customer/ other business teams and provide feedback on requirements