Data Engineer - Azure Databricks

Location: London - Hybrid (3 days per week in the office)

Salary: £90,000 - £100,000 + Benefits

We are looking for an experienced Senior Data Engineer to join a major organisation working on a large-scale Azure Databricks platform.

This is a hands-on technical role focused on building, developing and optimising complex data pipelines, improving data quality and supporting the ongoing development of a modern cloud-based data platform.

The successful candidate will have extensive experience in data engineering, strong Azure expertise and a proven background working with Databricks in complex enterprise environments.

Key Responsibilities

  • Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and SQL.
  • Build and optimise ETL/ELT processes, integrating data from APIs, databases and other external sources.
  • Work extensively with Databricks Unity Catalog, Delta Lake and Spark SQL.
  • Develop complex data transformations, ensuring data accuracy, consistency and performance.
  • Implement data quality checks, monitoring and validation processes.
  • Utilise Microsoft Purview for data governance, lineage tracking and metadata management.
  • Optimise Databricks workloads for performance, scalability and cost efficiency.
  • Develop and maintain CI/CD pipelines using Azure DevOps and other automation tools.
  • Collaborate with data architects, engineers and wider technical teams to deliver scalable data solutions.

Essential Skills & Experience

  • 10+ years of experience in Data Engineering.
  • Minimum 3+ years of hands-on Azure Databricks experience.
  • Strong programming skills in Python, PySpark and SQL.
  • Extensive experience with Azure Data Factory, Azure Blob Storage and Azure SQL Database.
  • Strong understanding of Databricks Unity Catalog, Delta Lake and Spark SQL.
  • Experience designing and optimising large-scale data pipelines.
  • Strong knowledge of data modelling, data warehousing and ETL/ELT processes.
  • Hands-on experience with Microsoft Purview and data governance.
  • Experience integrating data from APIs, relational databases and streaming sources.
  • Experience with Git, CI/CD pipelines and Azure DevOps.
  • Strong problem-solving skills and the ability to work in complex enterprise environments.

Desirable Experience

  • Databricks or Microsoft Azure certifications.
  • Experience with Kafka or Azure Event Hubs.
  • Knowledge of Scala, R, YAML or JavaScript.
  • Experience with Power BI or Tableau.
  • Exposure to Docker, Kubernetes or Jenkins.
  • Experience working with large, complex datasets in highly regulated environments.

Job Details

Company
83Zero Ltd
Location
City, London, United Kingdom EC1A2
Hybrid / Remote Options
Employment Type
Permanent
Salary
GBP 80,000 - 100,000 Annual
Posted