Data Engineer - Azure Databricks
Location: London - Hybrid (3 days per week in the office)
Salary: £90,000 - £100,000 + Benefits
We are looking for an experienced Senior Data Engineer to join a major organisation working on a large-scale Azure Databricks platform.
This is a hands-on technical role focused on building, developing and optimising complex data pipelines, improving data quality and supporting the ongoing development of a modern cloud-based data platform.
The successful candidate will have extensive experience in data engineering, strong Azure expertise and a proven background working with Databricks in complex enterprise environments.
Key Responsibilities
- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and SQL.
- Build and optimise ETL/ELT processes, integrating data from APIs, databases and other external sources.
- Work extensively with Databricks Unity Catalog, Delta Lake and Spark SQL.
- Develop complex data transformations, ensuring data accuracy, consistency and performance.
- Implement data quality checks, monitoring and validation processes.
- Utilise Microsoft Purview for data governance, lineage tracking and metadata management.
- Optimise Databricks workloads for performance, scalability and cost efficiency.
- Develop and maintain CI/CD pipelines using Azure DevOps and other automation tools.
- Collaborate with data architects, engineers and wider technical teams to deliver scalable data solutions.
Essential Skills & Experience
- 10+ years of experience in Data Engineering.
- Minimum 3+ years of hands-on Azure Databricks experience.
- Strong programming skills in Python, PySpark and SQL.
- Extensive experience with Azure Data Factory, Azure Blob Storage and Azure SQL Database.
- Strong understanding of Databricks Unity Catalog, Delta Lake and Spark SQL.
- Experience designing and optimising large-scale data pipelines.
- Strong knowledge of data modelling, data warehousing and ETL/ELT processes.
- Hands-on experience with Microsoft Purview and data governance.
- Experience integrating data from APIs, relational databases and streaming sources.
- Experience with Git, CI/CD pipelines and Azure DevOps.
- Strong problem-solving skills and the ability to work in complex enterprise environments.
Desirable Experience
- Databricks or Microsoft Azure certifications.
- Experience with Kafka or Azure Event Hubs.
- Knowledge of Scala, R, YAML or JavaScript.
- Experience with Power BI or Tableau.
- Exposure to Docker, Kubernetes or Jenkins.
- Experience working with large, complex datasets in highly regulated environments.