Senior Data Engineer

Senior Data Engineer / Data Platform Engineer (Databricks)

Contract | Outside IR35

Remote (UK/EU)

£550–£650 per day (DOE)

We’re partnering with a forward-thinking Data & AI consultancy that is helping to scale a Databricks Lakehouse platform.

They are looking for a Senior Data Engineer / Data Platform Engineer who can operate across both Data Engineering & Platform Engineering.

This is an end-to-end hands-on role.

What You’ll Be Doing:

• Designing and building end-to-end data pipelines using Databricks (PySpark / SQL / Delta Lake)

• Developing and managing robust, scalable Lakehouse architectures

• Implementing Delta Live Tables (DLT) for reliable, testable, production-grade pipelines

• Optimising performance using:

• Working with Unity Catalog to implement Centralised governance

• Leveraging Databricks Workflows / Jobs for orchestration

• Building CI/CD pipelines for data & platform deployments (Terraform, GitHub Actions, Azure DevOps etc.)

• Enabling real-time and streaming pipelines (Structured Streaming, Auto Loader)

• Contributing to MLOps workflows, integrating models into production pipelines

• Collaborating with Data Scientists, Analysts, and Platform teams to improve developer experience and platform usability

Key Skills & Experience:

• Strong experience building pipelines with PySpark / Spark SQL

• Deep understanding of Delta Lake

• Databricks Platform Expertise

• Hands-on experience with: Unity Catalog (governance & security), Spark Declarative Pipelines, Databricks Workflows / Jobs

• Strong understanding of Lakehouse architecture principles

• Performance & Optimisation

• Platform / DevOps Engineering, Infrastructure as Code, CI/CD pipelines

• Strong experience in at least one - Azure, AWS

Very Nice to Have:

• Databricks certifications (Data Engineer Associate / Professional)

• Experience with Feature Store / MLflow

• Exposure to Databricks AI / Mosaic AI / Model Serving

If you’re interested in building high-performance Databricks platforms at scale, send your CV to cv@primus-connect.com or click apply.