Lead Databricks Engineer
Location:
London; UKWork type: HybridSeniority: LeadPrimary skill: Data Software EngineeringPosted: 2026-09-29We're looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities.
You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment. ResponsibilitiesLead architecture design and implementation of scalable solutions on Azure DatabricksDefine and enforce platform standards, ensuring compliance with data governance and security policiesBuild and optimize Databricks streaming workloads, including Structured Streaming and Delta Live TablesCollaborate with cross-functional teams to translate requirements into robust technical solutionsDrive platform performance tuning, cost optimization and monitoring strategies for cloud workloadsProvide leadership across the full engineering lifecycle, including solution delivery and production supportMentor and guide developers, establishing best practices for data engineering on Azure and DatabricksParticipate in technical decision-making and review designs for scalability and maintainabilityImplement observability for critical pipelines and maintain quality assurance standardsEngage directly with stakeholders to ensure alignment on strategic platform initiativesRequirementsMinimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud-native engineering patternsProven experience implementing data governance, quality management and observability frameworksAbility to manage complex data pipelines for batch and streaming use cases in mission-critical environmentsStrong communication and leadership skills to work with distributed teams and senior stakeholdersNice to haveKnowledge of Delta Live Tables (DLT) and advanced workflows in DatabricksFamiliarity with CI/CD for data engineering using GitHub or Azure DevOpsExperience designing infrastructure-as-code solutions for Azure environmentsUnderstanding of data mesh or lakehouse principles for large-scale architecturesBackground in financial trading or capital markets data domainsWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness cover Private medical insurance and dental care Employee Assistance Program Competitive group pension plan Cyclescheme, Techscheme and season ticket loans Various perks such as free Wednesday lunch in-office, on-site massages and regular social eventsLearning and development opportunities including in-house training and coaching, professional certifications, and coursesIf otherwise eligible, participation in the discretionary annual bonus program If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program*All benefits and perks are subject to certain eligibility requirements