1 to 25 of 39 Permanent Delta Lake Jobs in London

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

scalable Lakehouse solutions supporting critical business data. What you'll be doing Leading Azure Databricks architecture Designing enterprise Lakehouse solutions Building scalable Spark and Delta Lake pipelines Designing streaming and batch data platforms Driving engineering best practice Mentoring engineers Working closely with senior business stakeholders Defining future-state … data architecture Technical Environment Azure Databricks Apache Spark/PySpark Delta Lake Azure Data Factory Azure Data Lake Gen2 Azure Event Hubs Azure Synapse Python SQL Terraform CI/CD Medallion Architecture We're looking for Expert Databricks experience Strong Azure Data Platform experience Enterprise Data Architecture ...

governance controls and operational guardrails Technical Leadership & Standards Serve as the senior technical authority for the Databricks platform architecture Establish engineering best practices covering Delta Lake design, workload orchestration, compute optimization, and cost management Provide technical assurance across platform implementation activities delivered by internal teams and external partners … communication, stakeholder engagement, and consulting capabilities Experience translating business requirements into strategic platform architecture solutions Technical Skills Deep expertise in Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and cluster management Strong experience designing cloud-native data platforms on AWS Hands‐on knowledge of AWS services including ...

enterprise-grade business insights. Key Responsibilities Data Architecture & CDL Governance: Design, implement, and govern the enterprise Common Data Layer (CDL) on Azure Databricks using Delta Lake and L0–L1–L2 architectures. Establish data quality frameworks, schema evolution standards, and deduplication logic. Pipeline & Integration Engineering: Oversee the creation … Databricks data assets into robust, secure, and intuitive Power BI dashboards. Required Experience & Skills Technical Expertise Cloud Data Platforms: Advanced knowledge of Azure Databricks, Delta Lake, Azure Data Factory (ADF), and SQL querying. SAP Ecosystem: Deep data discovery skills within SAP S/4HANA, SAP BW/HANA ...

risks, exceptions, and technical dependencies raised through governance processes Enterprise & Solution Architecture Provide architecture assurance for Databricks Lakehouse implementations, including data ingestion, medallion architecture, Delta Lake, Unity Catalog, and compute strategies Validate integration patterns across Databricks, cloud platforms, enterprise applications, and data ecosystems Define and maintain reference architectures … delivery requirements Exceptional stakeholder engagement and influencing skills across technical and business audiences Technical Skills Strong hands‐on knowledge of Databricks including Lakehouse Architecture, Delta Lake, Unity Catalog, and platform governance Experience with cloud platforms supporting Databricks environments, including AWS and/or Azure Solid understanding of enterprise ...

retry mechanisms. Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing. Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault. Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling … reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications. Experience with entity resolution, record matching platforms, and third-party enrichment services. Experience with Delta Lake and Change Data Feed (CDF) patterns. Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation. Experience ...

production experience is not required. A collaborative mindset and willingness to learn and communicate openly within the team. Nice to have Exposure to data lake concepts or formats such as Delta Lake or Iceberg. Any experience with search, graph, or large-scale analytical databases. Interest ...

delivering large scale Data science or AI Projects Strong knowledge of LLMs (GPT, Llama, Claude, Mistral) and their reasoning capabilities Deep Databricks experience (DLT, Delta Lake, Unity Catalog governance) Solid grasp of streaming technologies (Spark Structured Streaming, Autoloader) Python, SQL, or Scala proficiency Experience with ETL/ ...

data fidelity. Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. ...

design, workshops and Proof of Concepts. What you'll bring: Proven experience designing enterprise data platforms using Databricks . Strong knowledge of Spark, PySpark, Delta Lake, SQL and modern data engineering. Expertise with Microsoft Azure Strong understanding of data modelling, governance and ETL pipelines. Excellent stakeholder management ...

data science/AI projects. Deep understanding of LLM families (GPT, Llama,Claude, Mistral) and their reasoning capabilities. S trong experience with Databricks- DLT,Delta Lake concepts, UC governance. Solid understanding of streamingtechnologies (e.g., Spark Structured Streaming, Autoloader) Programming skills in Python, SQL, or Scala. Proficiency in data ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

legacy operational platforms. Experience with regulatory, client or operational reporting. Experience supporting data migration, legacy system modernisation or platform replacement. Experience with Databricks, Delta Lake, Spark SQL or PySpark. Experience defining validation frameworks, data quality dashboards or metadata-driven rule engines. Experience working in consultancy or client-facing ...

within Agile environments. Excellent communication and stakeholder management skills. Technical Environment Experience in some or all of the following is highly desirable: AWS Databricks Delta Lake Lakehouse architectures Complex SQL environments ETL/ELT pipelines Change Data Capture (CDC) Data lineage and metadata management tools Reporting and analytics ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn, and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits ...

performance, cost and standards-compliance metrics, using them to guide roadmap priorities and architecture governance decisions. PERSONAL PROFILE Deep expertise in Databricks (Unity Catalog, Delta Lake, Spark, Databricks SQL) or equivalent Lakehouse platforms, including administration, cost management, and performance optimisation. Strong understanding of cloud-native architecture principles, cost ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

AI. the company is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow the company on Twitter, LinkedIn and Facebook. At the company, we strive to provide comprehensive benefits ...

Must Haves (Hard Skills) Data Lakehouse Mastery: Proven experience building data infrastructure using Trino/Presto and open table formats like Apache Iceberg or Delta Lake. Advanced Data Modeling: Deep expertise in Medallion Architecture patterns, Star Schemas, and processing semi‐structured financial data logs (JSON/Avro). Robust ...

Must Haves (Hard Skills) Data Lakehouse Mastery: Proven experience building data infrastructure using Trino/Presto and open table formats like Apache Iceberg or Delta Lake. Advanced Data Modeling: Deep expertise in Medallion Architecture patterns, Star Schemas, and processing semi-structured financial data logs (JSON/Avro). Robust ...