1 to 25 of 36 Delta Lake Jobs in the UK

platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead … experience * Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

scalability, security, and governance. Act as Databricks champion: define standards, reference architectures, accelerators, and reusable assets. Lead architectural design for Databricks (workspaces, clusters, jobs, Delta Lake, Unity Catalogue, MLflow) and integrations with cloud services. Partner with pre-sales teams to shape propositions, run discovery, produce high-quality proposals … encryption, auditing, lineage). Essential Requirements Significant Databricks experience required, including production-scale deployments and optimisation. Strong hands-on capability with Spark/SQL, Delta Lake, orchestration, and data engineering patterns. Prior experience within a consulting practice with exposure to both pre-sales and delivery leading ...

engineering challenges, and driving best practice across the platform. What you'll be doing Designing, building and optimising Databricks pipelines and jobs Creating scalable Delta Lake solutions using Medallion architecture Developing and maintaining Azure Data Factory pipelines Owning data models across bronze, silver and gold layers Driving … OneLake environments What we're looking for Strong commercial experience with Databricks in production environments Excellent data modelling and architecture skills Expertise in Delta Lake and Medallion design principles Azure Data Factory experience, including parameterised and environment-driven pipelines Strong SQL skills plus Python and/or Scala ...

deliver actionable, enterprise-grade business insights.Key ResponsibilitiesData Architecture & CDL Governance: Design, implement, and govern the enterprise Common Data Layer (CDL) on Azure Databricks using Delta Lake and L0–L1–L2 architectures. Establish data quality frameworks, schema evolution standards, and deduplication logic.Pipeline & Integration Engineering: Oversee the creation of scalable … connection of Azure Databricks data assets into robust, secure, and intuitive Power BI dashboards.Required Experience & SkillsTechnical ExpertiseCloud Data Platforms: Advanced knowledge of Azure Databricks, Delta Lake, Azure Data Factory (ADF), and SQL querying.SAP Ecosystem: Deep data discovery skills within SAP S/4HANA, SAP BW/HANA ...

/ELT solutions and enterprise data pipelines. Experience with event-driven and messaging architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture ...

including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices (testing ...

Strong Azure Data Factory (ADF) and Azure Databricks experience. Experience building ETL/ELT pipelines within Azure Lakehouse environments. Knowledge of Medallion Architecture and Delta Lake. Strong SQL and Python skills. Experience with Power BI datasets and data modelling. Familiarity with Azure DevOps, Git, CI/CD and Infrastructure ...

remaining actively involved in hands-on delivery. Experience working across cross-functional teams including Architecture, Product, Engineering, and Business stakeholders. Data Engineering & Platforms Databricks Delta Lake Unity Catalog dbt Spark/PySpark Data pipeline development and optimisation Lakehouse architecture Data integration and ETL/ELT frameworks Data platform ...

such as Snowflake's Snowpark, Dynamic Tables, Streams & Tasks, and Snowflake Cortex, as well as Databricks components such as Lakeflow (Connect, Pipelines, and Jobs), Delta Lake, and Unity Catalog. Strong data engineering fundamentals: ETL/ELT pipeline design and implementation, data orchestration and workflow automation, batch and streaming ...

pipeline development, and large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security … Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most prominent corporations ...

PySpark GitLab/Azure DevOps pipelines Experience validating large-scale datasets and data transformations Knowledge of data warehousing concepts and dimensional modeling Understanding of Delta Lake, Parquet, and Lakehouse architectures Experience with CI/CD and automated deployment validation Familiarity with Agile/Scrum delivery methodologies Microsoft Fabric ...

accelerate delivery, and enhance solution quality while applying sound engineering judgement and responsible AI practices.What You'll DoDesign and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services.Build real-time, event-driven solutions using technologies such as Kafka and Azure Event ...

improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration patterns for REST … Azure Databricks implementation and troubleshooting Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration ...

planning, solution design and adoption. Data Engineering and Integration Design, build and optimise scalable data ingestion and transformation solutions using Python, PySpark, SQL and Delta Lake. Implement batch and incremental processing patterns, including Change Data Capture (CDC), schema evolution, reconciliation processes, data quality controls and error handling. Develop reusable … administration and troubleshooting experience. Databricks Serverless architecture, workload placement and compute optimisation. Azure networking, identity, security, monitoring, secrets management and private connectivity. Databricks SQL, Delta Lake and query/workload performance optimisation. Strong understanding of platform governance, operational support and best practices. Highly Desirable Experience migrating workloads from ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

secure handling of inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

delivering large scale Data science or AI Projects Strong knowledge of LLMs (GPT, Llama, Claude, Mistral) and their reasoning capabilities Deep Databricks experience (DLT, Delta Lake, Unity Catalog governance) Solid grasp of streaming technologies (Spark Structured Streaming, Autoloader) Python, SQL, or Scala proficiency Experience with ETL/ ...

workflows and automated testing. Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability. The Individual Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow. Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral. Experience building and deploying GenAI ...

workflows and automated testing. Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability. The Individual Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow. Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral. Experience building and deploying GenAI ...

workflows and automated testing. Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability. The Individual Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow. Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral. Experience building and deploying GenAI ...

workflows and automated testing. Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability. The Individual Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow. Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral. Experience building and deploying GenAI ...

ability to work effectively with both technical and non-technical stakeholders Desirable Experience Experience with any of the following would be highly beneficial Delta Lake Unity Catalog MLfow Databricks optimisation and performance tuning AI, Machine Learning or Generative AI initiatives AI-assisted development tools such as Claude, GitHub ...