Senior Data Engineer

Senior Data Engineer - Hybrid - Contract - Outside IR35

Our client is expanding their core data engineering function and hiring two Senior Data Engineers to strengthen a modern, scalable data platform. Both will be Databricks specialists, with one role focused on deep technical delivery and the other combining hands-on engineering with team leadership and upskilling responsibilities.

Key Responsibilities

• Build and maintain Databricks Delta Live Tables (DLT) pipelines across Bronze → Silver → Gold layers, ensuring quality, scalability, and reliability.
• Develop and optimise Spark (PySpark) jobs for large-scale distributed processing.
• Design and implement streaming data pipelines with Kafka/MSK, applying best practices for late event handling and throughput.
• Use Terraform and CI/CD pipelines (GitHub Actions or similar) to manage infrastructure and automate deployments.
• (For leadership-focused role) Mentor and upskill engineers, define coding standards, and embed engineering excellence across the team.

What’s Expected

• Proven experience delivering end-to-end data pipelines in Databricks and Spark environments.
• Strong understanding of data modelling, schema evolution, and data contract management.
• Hands-on experience with Kafka, streaming architectures, and real-time processing principles.
• Proficiency with Docker, Terraform, and cloud platforms (AWS, GCP, or Azure) for scalable data infrastructure.
• Demonstrated ability to lead, coach, and elevate team capability, fostering best practices and knowledge sharing.