Databricks Solution Architect/Lead

JOB DESCRIPTION SUMMARY We are seeking a Databricks Solution Architect/Team Lead to join our Medical Insurance Client/Implementation Partner team. The ideal candidate will have a proven track record as a senior/self-starting data engineer in implementing data ingestion and transformation pipelines for large scale organizations. We are seeking someone with technical skills in Databricks development, performance tuning, and optimization. The candidate will assist in the design and development of high performant data ingestion pipelines from multiple sources using Databricks. The candidate will be involved in all stages of integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained. Candidate will have extensive experience with commercial and open source relational and non-relational data repositories. Develop scalable and re-usable frameworks for ingestion and transformation of large data sets. Design and implement data ingestion pipelines from multiple sources ensuring the quality and consistency of data is always maintained. Work with event based / streaming technologies to ingest and process data. Work with other members of the project team to support delivery of additional project components (API interfaces, Search). Stream and Batch processes in Databricks. Work within an Agile delivery / Dev Ops methodology to deliver proof of concept and production implementation in iterative sprints. Convert existing Informatica ETL code to Databricks code whenever it is feasible/required. Create and maintain Databricks queries to support dashboarding and/or reporting activities. SQL query development and optimization as required to support various reporting needs. Performance monitoring and diagnosis of the ingest pipeline and suggest continual improvement. Position is contingent on funding. Project Specific Qualifications: Bachelor's Degree from an accredited college or university required; an additional four (4) years of related work experience can substitute for a degree At least five (5) years of relevant experience required At least two (4) years of experience with Databricks. Expertise in designing and deploying data applications on cloud solutions, such as Azure or AWS Comprehensive understanding of data management best practices including demonstrated experience with data profiling, sourcing, and cleansing routines utilizing typical data quality functions involving standardization, transformation, rationalization, linking and matching. Experience in building ETL / data warehouse transformation processes Hands on experience in performance tuning and optimizing code running in programming languages such as PySpark and Python Good understanding of SQL, T-SQL and/or PL/SQL Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment Experience with Apache Kafka for use with streaming data / event-based data Experience with other Open-Source big data products Hadoop (incl. Hive, Pig, Impala) Experience working with structured and unstructured data Experience working in an Agile Dev Sec Ops environment Experience of working with relational databases: (SQL Server, Postgre SQL) Experience with non-relational / No SQL data repositories (incl. Mongo DB, Cassandra, Neo4J) Preferred experience: Knowledge of IRS business systems and data Experience working in a command line environment and a general understanding of Red Hat Linux OS, or other Unix-like OS. Databricks certification Expertise with creating custom visualizations (e.g. implementing force-directed graph visualization using D3.js) for Tableau, Power BI, etc. Expertise with ELK stack and/or Splunk.Essential Duties and Responsibilities: Lead the development of software solutions that will meet or exceed business requirements; the development effort includes designing and implementing modules to the system specifications, conducting unit testing, troubleshooting issues and producing detailed proposals to resolve issues. May evaluate new coding techniques, tools, modules, and implementation as appropriate. Lead and mentor entry and mid-level developers. Consult on requirements elicitation and definition. Design software solutions per systems requirements. Code software solutions per designs. Code reviews, unit test, and integrate coded modules. Assist other developers in resolving issues by providing guidance and training. Support testing and remediate defects.