Lead Data Engineer - Databricks

Lead Data Engineer - Databricks

Location: Remote UK or hybrid / onsite at one of our clients’ offices

Working Pattern: Remote UK / Hybrid or office based

Salary: Competitive

ARCA Resourcing is proud to be partnered with an innovative and leading retailer that is investing heavily in its data capabilities to drive smarter decision-making across the business. As part of this transformation, we’re looking for a Lead Data Engineer to play a key role in shaping and delivering the organisation’s modern Enterprise Data Platform.

This is an opportunity to combine hands-on engineering with technical leadership, working with cutting-edge technologies while mentoring engineers and influencing the direction of a high-impact data platform.

The Opportunity

The organisation operates a modern data stack built around a Databricks-based lakehouse architecture, alongside a Customer Data Platform (CDP), enterprise analytics tooling, and self-service reporting capabilities.

As a Lead Data Engineer, you will oversee multiple delivery squads, ensuring consistent engineering standards and guiding the design and delivery of scalable data solutions. You’ll coordinate technical delivery across teams while remaining hands-on with architecture, pipelines, and best practice development.

This role is ideal for someone who enjoys leading teams, shaping engineering standards, and building high-performance data platforms at scale.

Key Responsibilities

• Provide technical leadership across multiple delivery squads, ensuring consistent data engineering standards and patterns.
• Coordinate technical solutions and align team efforts across initiatives.
• Translate functional designs into scalable and reusable data engineering solutions.
• Collaborate closely with solution architects, business analysts, and stakeholders to refine requirements and technical approaches.
• Design and implement high-performance ELT pipeline patterns using Databricks.
• Mentor and develop engineers across the team, creating learning and development plans.
• Champion engineering best practices, including code quality, testing, and documentation.
• Conduct peer reviews to ensure high-quality and reliable deliverables.
• Optionally contribute to on-call platform support where required.

What We’re Looking For

Technical Expertise

• Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure).
• Excellent programming capability in Python and SQL.
• Experience orchestrating pipelines with Azure Data Factory and Databricks Workflows.
• Solid understanding of Lakehouse architectures and modern data warehousing principles.
• Experience with data modelling, ELT/ETL pipelines, and testing frameworks.
• Familiarity with CI/CD pipelines, version control, and DevOps practices.
• Knowledge of security, governance, and cost optimisation in cloud data platforms.

Leadership Experience

• Proven experience leading engineering teams or coordinating multiple squads.
• Experience aligning engineering work with strategic project objectives.
• Ability to mentor and develop engineers, driving technical excellence across teams.
• Strong problem-solving, debugging, and architectural thinking skills.

Desirable

• Degree in IT, Computer Science, or related discipline.

If this opportunity is of interest, please apply for immediate consideration!