Cloud Data Engineer

05 Nov 2024

Vacancy expired!

Business Intelligence/Data Visualization EngineerPartially OnsiteAbout the Job

  • Duration: Long term renewable contract
  • Location: Greenville, SC
  • Pay rate: Hourly, depending on experience
  • Job ID: 3952
This is a Full-Time W2 position.During initial 2 weeks onboarding period would need to be 100% onsite.Onsite would be 2 days every other week.If required, the candidate may have to travel up to 5-10%OverviewDatasoft Technologies is looking for a Cloud Data Engineer who can provide complete application lifecycle development, deployment, and operations support for large-scale batch and real-time data processing pipelines using cloudtechnologies. Who collaborates with product owners, data scientists, business analysts, and software engineers to design and build solutions to ingest, transform, store, and export data in a cloud environment while maintaining security, scalability, and personal data protection. The day-to-day tasks would be working on ingestion jobs and use cases to understand the business logic that needs to happen for the data to make sense from a business perspective.There is a lot of development with Python and an understanding of Terraform and Glue and Athena within AWS.Responsibilities
  • This position is with the advanced data analytics team. They are responsible for all of the data in the data lake which is moved into AWS. They are responsible for the ingestion work and all the transformations that happen with the data sets to make them consumable for use cases.
  • Implements and enhances complex data processing pipelines with a focus on collecting, parsing, cleaning, managing, and analyzing large data sets that produce valuable business insights and discoveries.
  • Determines the required infrastructure, services, and software required to build advanced data ingestion & transformation pipelines and solutions in the cloud.
  • Assists data scientists and data analysts with data preparation, exploration, and analysis activities.
  • Applies problem-solving experience and knowledge of advanced algorithms to build high-performance, parallel, and distributed solutions.
  • Performs code and solution review activities and recommends enhancements that improve efficiency, performance, stability, and decreased support costs.
  • Applies the latest DevOps and Agile methodologies to improve delivery time.
  • Works with SCRUM teams in daily stand-up, providing progress updates frequently.
  • Supports application, including incident and problem management.
  • Performs debugging and triage of incidents or problems and deployment of fix to restore services.
  • Documents requirements and configurations and clarifies ambiguous specs.
  • Performs other duties as assigned by management.
  • This role critical requires Python and understanding of Terraform and Glue and Athena within AWS.
Qualifications
  • BA/BS Degree in Business, Computer Science, or Electrical Engineering preferred or the equivalent of 4 years professional IT-related experience. MS degree (preferred).
Experience :
  • 3+ years of enterprise software engineering experience with object-oriented design, coding, and testing patterns, as well as, experience in engineering (commercial or open-source) software platforms and large-scale data infrastructure solutions.
  • 3+ years of software engineering and architecture experience within a cloud environment (Azure, AWS).
  • 3+ years of enterprise data engineering experience within any "Big Data " environment (preferred).
  • 3+ years of software development experience using Python.
  • 2+ years of experience working in an Agile environment (Scrum, Lean, or Kanban).
  • 3+ years of experience working in large-scale data integration and analytics projects, including using cloud (e.g. AWS Redshift, S3, EC2, Glue, Kinesis, EMR) and data-orchestration (e.g. Oozie, Apache Airflow) technologies
  • 3+ years of experience in implementing distributed data processing pipelines using Apache Spark
  • 3+ years of experience in designing relational/NoSQL databases and data warehouse solutions
  • 2+ years of experience in writing and optimizing SQL queries in a business environment with large-scale, complex datasets
  • 2+ years of Unix/Linux operating system knowledge (including shell programming).
  • 1+ years of experience in automation/configuration management tools such as Terraform, Puppet, or Chef.
  • 1+ years of experience in container development and management using Docker.
  • Languages: SQL, Python , Spark
  • Basic knowledge of continuous integration tools (e.g. Jenkins).Basic knowledge of machine learning algorithms and data visualization tools such as Microsoft Power BI and Tableau.
  • Experience with S3, Python, and understanding of Terraform and Glue and Athena within AWS is preferred.
  • Experience with Spark, Hadoop, Linux would be preferred.
  • Certifications/Skills: AWS/Azure cloud provider training/certifications (preferred)
  • Python is a must-have. AWS experience is strongly preferred.
  • Development experience is also a must.
About our Company DataSoft Technologies is a highly recognized provider of professional IT Consulting services in the US. Founded in 1994, DataSoft Technologies, Inc. provides staff augmentation services for Information Technology and Automotive Services. Our team memberbenefits include:
  • Paid Holidays/Paid Time Off (PTO)
  • Medical/Dental Insurance
  • Vision Insurance
  • Short Term/Long Term Disability
  • Life Insurance
  • 401 (K)

  • ID: #22213751
  • State: South Carolina Greer 29651 Greer USA
  • City: Greer
  • Salary: BASED ON EXPERIENCE
  • Job type: Permanent
  • Showed: 2021-11-05
  • Deadline: 2022-01-02
  • Category: Et cetera