Databricks Pipeline Engineer

I am recruiting for a Databricks Pipeline Engineer to work on a remote basis.

The role falls inside IR35 so you will be required to work through an umbrella company for the duration of the contract.

Strong expertise in Databricks Pipelines.

You must have experience of designing, optimising and maintaining scalable Databricks pipelines (ETL, streaming, ML workflows).

You will be able to perform cluster and job performance tuning: optimise cluster sizing, caching, partitioning, and shuffle management.

You will monitor Spark job metrics, analyse logs, and identify bottlenecks in data throughput or latency.

You will also be able to implement cost-optimisation strategies for Databricks jobs and clusters using autoscaling and job consolidation.

You must have familiarity with orchestration tools: Databricks Workflows, Airflow, or Azure Data Factory.

Proficiency in Python or Scala for data engineering and pipeline development is essential.

Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.

Knowledge of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs.

Experience of version control (Git, GitHub Actions, DevOps pipelines) and CI/CD practices for Databricks.

Please apply ASAP to find out more.