Lead Databricks Engineer

Back to SearchLooking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a jobWe're looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities.

You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment. Lead architecture design and implementation of scalable solutions on Azure DatabricksDefine and enforce platform standards, ensuring compliance with data governance and security policiesBuild and optimize Databricks streaming workloads, including Structured Streaming and Delta Live TablesCollaborate with cross-functional teams to translate requirements into robust technical solutionsDrive platform performance tuning, cost optimization and monitoring strategies for cloud workloadsProvide leadership across the full engineering lifecycle, including solution delivery and production supportMentor and guide developers, establishing best practices for data engineering on Azure and DatabricksParticipate in technical decision-making and review designs for scalability and maintainabilityImplement observability for critical pipelines and maintain quality assurance standardsEngage directly with stakeholders to ensure alignment on strategic platform initiativesMinimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud-native engineering patternsProven experience implementing data governance, quality management and observability frameworksAbility to manage complex data pipelines for batch and streaming use cases in mission-critical environmentsStrong communication and leadership skills to work with distributed teams and senior stakeholdersKnowledge of Delta Live Tables (DLT) and advanced workflows in DatabricksFamiliarity with CI/CD for data engineering using GitHub or Azure DevOpsExperience designing infrastructure-as-code solutions for Azure environmentsUnderstanding of data mesh or lakehouse principles for large-scale architecturesBackground in financial trading or capital markets data domains