Databricks Specialist

Databricks Platform Specialist | Permanent opportunity

Role Overview

We are seeking a highly skilled Databricks Platform Specialist to take ownership of a cloud-based Databricks environment, ensuring it is secure, governed, scalable, and optimised to support data, analytics, and AI initiatives across the organization.

This role sits at the intersection of platform engineering, data governance, and cloud operations. The successful candidate will act as the subject matter expert for the Databricks platform, overseeing administration, governance, engineering standards, and operational excellence while supporting the adoption of advanced analytics and AI capabilities.

Working closely with data engineers, architects, infrastructure teams, security teams, and business stakeholders, you will play a key role in ensuring the platform delivers reliable, high-quality services and supports strategic business objectives.

Key Responsibilities

Platform Administration & Governance

• Own the day-to-day administration of the Databricks environment, including workspace configuration, cluster policies, and instance pool management.
• Administer Unity Catalog as the central governance layer, managing metastores, catalogs, schemas, and access controls.
• Define and enforce compute governance standards, including cluster sizing, auto-termination policies, and cost optimization controls.
• Manage platform upgrades, Databricks Runtime versioning, and library dependency standards.
• Maintain platform monitoring, alerting, and incident management processes, acting as the primary escalation point for platform-related issues.

Security & Access Management

• Design and maintain role-based access control (RBAC) models aligned to data governance and security requirements.
• Work closely with security and infrastructure teams to ensure compliance with organizational standards and regulatory requirements.
• Manage service principals, access tokens, credential management, and identity integrations.
• Support audit logging, lineage tracking, and regular access reviews.

Data Engineering Standards & Governance

• Establish and maintain engineering standards, notebook templates, and development patterns for Delta Lake workloads.
• Review data pipeline designs and promote best practices across engineering teams.
• Lead the configuration and governance of self-service analytics capabilities, ensuring appropriate controls, access management, and data quality standards.
• Support the implementation and governance of AI-enabled capabilities, model serving, and orchestration frameworks.

Infrastructure & DevOps

• Manage Databricks infrastructure using Infrastructure-as-Code principles and Terraform.
• Maintain and enhance CI/CD pipelines for Databricks asset deployment, including notebooks, jobs, and data pipelines.
• Collaborate with cloud infrastructure teams on networking, storage, identity management, and platform integration.

Platform Support & Enablement

• Provide technical guidance and support to teams using the platform.
• Assist with solution design and implementation to ensure alignment with governance standards and architectural best practices.
• Support onboarding activities and help teams adopt platform capabilities effectively.

Technical Leadership

• Act as the Databricks subject matter expert across the organization.
• Develop and maintain technical documentation, operational runbooks, governance standards, and onboarding materials.
• Stay current with developments in Databricks and cloud data technologies, evaluating new features and capabilities.
• Share knowledge through mentoring, technical reviews, and enablement activities.
• Build strong relationships with technology partners and vendors to maximize platform value and adoption.

Essential Skills & Experience

• 5+ years of hands-on Databricks experience in an enterprise environment.
• Strong expertise in Unity Catalog administration, including metastore management, catalog and schema design, access controls, and lineage.
• Advanced knowledge of Delta Lake, including optimisation, schema evolution, time travel, and medallion architecture.
• Experience using Terraform for Databricks Infrastructure-as-Code.
• Strong experience with CI/CD pipelines and automated deployment processes.
• Hands-on experience with Azure-hosted Databricks environments.
• Knowledge of cloud networking, identity management, storage services, and security best practices.
• Strong understanding of data governance, access management, audit logging, and compliance requirements.
• Proficiency in PySpark and SQL within production Databricks environments.
• Excellent communication and stakeholder management skills.

What We're Looking For

We are looking for an experienced Databricks professional who combines deep technical expertise with strong governance, operational, and stakeholder management skills. You will be passionate about building secure, scalable, and efficient data platforms while enabling teams to maximise the value of data, analytics, and AI technologies.