Hadoop Architect

28 Oct 2024

Vacancy expired!

Hadoop Architect Location: Charlotte, NC / Pennington, NJ / Dallas, TX Fulltime

  • Minimum 12+ years of experience. Be responsible for the design and support of the API(s) between IBM Spectrum Conductor (as a distributed compute and storage platform), and the Bank of America application that exposes a data scientist user experience and model governance.
  • Capture cluster tenant compute and storage functional and nonfunctional requirements; and translate into distributed cluster capacity, configuration, and user provisioning settings.
  • Develop, test, and analyze code/scripts written in PySpark, Python, Java, and other shell scripts, to provide specified behavior on a distributed IBM Spectrum Conductor cluster.
  • Provide "how-to" technical support for tenant developers developing runtimes and persisting data on a distributed IBM Spectrum Conductor cluster.
  • Be an active member of the Agile scrum and be a part of the features that emerge from the team.
  • Perform peer code and test case reviews and help foster a healthy technical community by helping peers.
  • Qualifications:
  • Experience with Agile/Scrum methodology is essential.
  • Experience with either Apache Spark-On-YARN (Hadoop) or Apache Spark-On-EGO (IBM Spectrum Conductor) is essential.
  • Experience with Apache Spark Libraries: PySpark, Spark SQL, Spark Streaming, MLlib are essential.
  • Experience with either Hadoop/YARN or IBM Spectrum Conductor/EGO cluster resource manager is essential.
  • Experience with RedHat Linux (RHEL) command line and shell scripting are essential.
  • Experience with file formats CSV, JSON, ORC, Avro, Parquet, Protocol Buffers are essential.
  • Experience with Python, Java, and R are highly desirable.
  • Experience with Numpy, and Pandas are highly desirable.
  • Experience with designing and configuring distributed architectures are desirable.
  • Knowledge of CI/CD SDLC practices.
  • Knowledge of Scikit-Learn, PyTorch, Keras, H20.ai.
  • Strong communication skills, should be able to communicate effectively with business and other stakeholders. Demonstrate ownership and initiative taking

  • ID: #21819260
  • State: Delaware Charlotte 28246 Charlotte USA
  • City: Charlotte
  • Salary: USD TBD TBD
  • Job type: Permanent
  • Showed: 2021-10-28
  • Deadline: 2021-12-26
  • Category: Et cetera