Databricks specialist

Databricks Specialist

Data, Analytics & AI | Senior Individual Contributor

Location: Manchester(preferred) And London 3 days onsite

Shape the Future of Enterprise Data & AI

Perm role

60000-80000 annually

We are looking for an experienced Databricks Specialist to become the technical owner of a large-scale Azure Databricks platform at the heart of a rapidly growing Data, Analytics & AI practice.

This is a high-impact opportunity for a Databricks expert who enjoys combining platform engineering, data governance, cloud architecture, and AI innovation. You will play a pivotal role in driving the adoption of modern data engineering standards, enterprise governance frameworks, and next-generation AI capabilities across both internal platforms and client-facing solutions.

Working closely with architects, data engineers, security teams, and business stakeholders, you will act as the Databricks Subject Matter Expert, ensuring the platform remains secure, scalable, cost-efficient, and aligned with strategic business objectives.

What You'll Be Doing

Databricks Platform Ownership

• Lead administration and optimisation of Azure Databricks environments.
• Manage workspace configuration, cluster policies, instance pools, and runtime upgrades.
• Define and implement platform governance, performance standards, and cost controls.
• Monitor platform health, performance, and operational stability.

Unity Catalog & Data Governance

• Own Unity Catalog administration including metastores, catalogs, schemas, and access controls.
• Design enterprise-grade governance frameworks and RBAC models.
• Support audit logging, lineage tracking, compliance reporting, and data access reviews.
• Ensure governance standards align with security and regulatory requirements.

Data Engineering Standards

• Champion Delta Lake best practices and Medallion Architecture principles.
• Define reusable engineering patterns, notebook standards, and deployment frameworks.
• Review solution designs and provide architectural guidance to delivery teams.
• Drive consistency and quality across all Databricks implementations.

AI & Analytics Enablement

• Lead the implementation and governance of Databricks Genie for conversational analytics.
• Support AI orchestration capabilities including Model Serving, Agent Framework deployments, and integrations with external AI services.
• Establish guardrails and governance frameworks for responsible AI adoption.

Infrastructure & DevOps

• Manage Databricks infrastructure using Terraform and Infrastructure-as-Code principles.
• Enhance CI/CD deployment pipelines through Azure DevOps.
• Collaborate with Azure infrastructure teams on networking, security, storage, and identity integration.
• Drive automation and operational excellence across the platform.

Technical Leadership

• Act as the Databricks SME across the organisation.
• Mentor engineers and support architectural decision-making.
• Produce technical documentation, governance standards, and operational runbooks.
• Evaluate emerging Databricks and Azure capabilities for future adoption.
• Partner directly with Databricks to maximise platform value, training, and enablement opportunities.

What We're Looking For

Essential Experience

• 5+ years of hands-on Databricks experience within enterprise environments.
• Strong expertise in Unity Catalog administration and governance.
• Deep knowledge of Delta Lake and Medallion Architecture.
• Experience with Terraform and Infrastructure-as-Code.
• Azure DevOps CI/CD pipeline implementation and management.
• Azure Databricks platform administration including networking, Entra ID, and ADLS Gen2.
• Strong understanding of data security, RBAC, audit logging, and compliance frameworks.
• Production-scale PySpark and SQL development experience.
• Excellent stakeholder engagement and communication skills.

Desirable Experience

• Professional Services or Consulting background.
• Microsoft Fabric, Azure Data Factory, Synapse Analytics, or Event Hubs.
• dbt on Databricks.
• MLflow and Databricks Asset Bundles.
• Data Mesh architecture principles.
• Databricks Certified Data Engineer Professional certification.
• FinOps and cloud cost optimisation experience.

Why Join?

• Own and shape a strategic enterprise Databricks platform.
• Work on cutting-edge Data, Analytics, AI, and Agentic AI initiatives.
• Influence platform strategy and architecture decisions.
• Collaborate with highly skilled engineers, architects, and AI specialists.
• Gain exposure to enterprise-scale client and internal transformation programmes.
• Continuous learning, certification, and career growth opportunities.