The Senior DevOps Engineer with Databricks Platform

ResponsibilitiesAs a Senior Software Engineer, Databricks Platform, your primary responsibilities will be to deploy and automate, administer, and monitor the core platform services provided by our AWS-hosted Databricks unified analytics platform• Serve as the primary “Databricks account owner”, working with our platform operations staff to oversee billing, subscription levels, workspaces, host AWS accounts, audit logs, and high-level usage monitoring of the hundreds of users on the platform• Work closely with the Privacera/Ranger DevOps Engineer on data access governance policy administration• Work closely with our AWS cloud devops team on compute, storage and network resource management, ensuring that Corning has a comprehensive view of resource consumption across AWS and Databricks• Collaborate closely with our IDP team to ensure proper operation of our SAML and SCIM identity management integrations• Work to implement Databricks delegated account administration with the various divisions and functions across the company• Serve as the primary Databricks workspace administrator and work to build out a team of workplace admins across the company• In all admin use cases, leverage the Databricks platform APIs to build out the capability to work through IaC automation with Terraform• Provide primary platform technical assist to our Qlik CDC Data Factory team and other teams building data engineering and ML engineering pipelines throughout Corning• Work collaboratively and proactively with software engineers and site reliability engineers across the platform engineering organization and the user community to anticipate, troubleshoot and resolve Databricks performance, connectivity, and security issues as they present themselves• Research, adopt and relate new technologies to solve issues that arise while adhering to company security and standards. Education & Experience• Undergraduate degree in Computer Science, IT, IS, or equivalent area of technical study• At least 12-18 months’ experience managing a production Databricks deployment as an account owner and workspace admin• 3 years’ builder experience on AWS, preferably with recent certification as solution architect, devops engineer• 3-5 years of hands-on production programming experience with agile software development in a variety of languages, such as Python, .NET/C#, Go, Java, JavaScript/Node.JS• 1-2 years of production Terraform IaC development and deployment experience• 3 years of production site reliability engineering experience across both cloud and on-premise environments Technical Qualifications:A minimum of 2 years hands-on staging and production experience in each of the following areas• Working with a team of 5 or more software engineers using Gitlab, Jenkins, or Azure DevOps.• Infrastructure and configuration management tools incliuding Terraform, Ansible or Chef• Container orchestration services, especially Docker & Kubernetes• Systems administration scripting methods such as Python, Bash or PowerShell. In-depth operational expertise in the following areas:• Cloud architecture best practices around operational excellence, security, reliability, performance efficiency, and cost optimization (e.g. AWS Well Architected Framework)• Best practices and IT operations for dynamic, always-up, always-available services Other Qualifications:• Demonstrated systems perspective when analyzing problems, thinking about overall operation, failure modes and how to address these problems proactively• A strong sense for the importance of documentation, and the importance of not having to learn things twice• Ability to work in an agile product team environment and balance a diverse set of stakeholder requests• Excellent oral and written communication skills with an ability to break down complex technical systems to help business partners understand the value• Ability to collaborate with other teams across the company, defining technology roadmaps, sharing experiences and lessons learned for continual improvement• Excellent problem-solving and troubleshooting skills• Process-oriented with great documentation skills