Data Engineer -Senior

25 Jul 2024

Vacancy expired!

Role: Data Engineer

Location: Dallas, TX (onsite / weekly, 2-3 days onsite)Detailed Activities
  • Discovery and Documentation
    • Analyze and document the ingestion tables and SAP Data Services Pipelines
    • Analyze and document the SQL Stored procedures for the Right-Angle tables
    • Document the .Net Process that makes updates into the Azure SQL DB
    • Document the business processes that happen in the DataBricks
    • Prepare the user stories for Data Ingestion and Preparation
    • Prepare a detailed schedule of activities for the next phases based on the information gathered in the Discovery phase
  • Data Preparation (Phase-1)
    • 40 Tables (approx.) needs to be ingested from Right Angle DB
    • SAP Data Services Transformation logics needs to be re-implemented in Azure
    • Remove the SAP Data Services dependency for Risk Warehouse
    • Utilize Azure Copy to move the data from On-Prem to the Azure Storage Account.
    • Adjusting the .Net app to utilize Azure Synapse Tables
  • Three-stage Code Migration (Phase-2)
  • Define Architecture and Create Connection to source and complete One Time Load Process including Alerts
    • Redevelop the Synapse pipelines with Spark for the Ontime load process as required
    • Create the Logic App for the alerts to be sent
    • Validate codes and data sets against the current production system
  • Daily Process – Synapse Pipeline Generation
    • Continue working on the Pipelines in Synapse with Spark for the Daily process
    • Create the Snapshot process using Spark.
    • Convert the functions that are utilized in DataBricks to Spark
    • Adjustment Process – Synapse Pipeline Generation (including Adjusting the .Net app to utilize Azure Synapse data)
      • Implement the .Net Application to allow the data writes to Synapse
      • Create the Utilities pipelines that are required from ADF to Synapse
      • Validate all the mappings and process that are migrated to the Synapse pipelines
    • Production Readiness
      • Validate the code against the Synapse platform.
      • Work with Business Users to test the application and data
      • Validate .Net User input and update and validation process
      • Validate the Data Snapshot process

    • ID: #44356123
    • State: Texas Dallas / fort worth 75201 Dallas / fort worth USA
    • City: Dallas / fort worth
    • Salary: Depends on Experience
    • Job type: Contract
    • Showed: 2022-07-25
    • Deadline: 2022-09-18
    • Category: Et cetera