Databricks Engineer

6-MONTH CONTRACT

£450-£550 PER DAY (OUTSIDE IR35)

This role is a great opportunity for a skilled Databricks Engineer to join a data-driven financial services firm undergoing a major transformation of their data infrastructure. You'll be a key player in modernising their data platform with Databricks and Azure, while supporting scalable data pipelines and machine learning workflows.

This business is investing heavily in data as it digitises operations across its risk, product, and customer intelligence teams. They're building a modern Lakehouse architecture using Databricks, Delta Lake, and Azure Data Lake. You'll be part of a collaborative, forward-thinking team with a focus on best practices, automation, and end-to-end data enablement.

You'll work closely with Data Engineers, Scientists, and Architects to design, build, and optimise core data pipelines on Databricks. Your focus will be on enabling analytics and machine learning at scale, using best-in-class tools across the Azure stack.

Your responsibilities will include:

• Designing and deploying ETL pipelines using PySpark and Delta Lake on Databricks.

• Supporting the deployment and operationalisation of ML models with MLflow and Databricks Workflows.

• Building out reusable data products and feature stores for data science teams.

• Tuning performance across clusters, jobs, and workflows.

• Migrating legacy systems (SSIS/SQL) to Databricks and cloud-native tools.

• Collaborating with data scientists, analysts, and product teams to improve data usability and performance.

• Advanced experience with Databricks, Delta Lake, and PySpark.

• Strong background in data engineering and distributed processing.

• Hands-on knowledge of Azure Data Lake, Data Factory, or similar orchestration tools.

• Experience with ML model deployment, preferably using MLflow or similar tools.

• Proficient in SQL, Python, and cloud-based data pipelines.

• Comfortable in fast-paced, agile delivery environments.

• Experience building or integrating feature stores.

• Familiarity with Unity Catalog, Databricks SQL, and cost optimisation.

• Exposure to MLOps practices and production-grade model lifecycle management.

• Prior experience in financial services or other regulated sectors.

Please register your interest by sending your CV via the apply link on this page. For more information, feel free to reach out directly.