1 to 25 of 60 Delta Lake Jobs in the UK

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted data at their fingertips. … governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop robust Bronze, Silver ...

deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products.Job responsibilitiesDesign and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost … pipelines on Databricks and/or Apache Spark in production.Strong coding skills in Python (PySpark) and SQL (Scala a plus).Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

pipelines that transform scientific datasets into analysis-ready formats aligned with FAIR data principles.Lead Databricks integration within Product Safety, driving best practices across Delta Lake, Unity Catalog, and data governance.Deliver strategic data product prototypes and partner with the Safety and Regulatory Data Domain Lead to transition these into … strong data engineering skills with the ability to work fluently across scientific and technical domains.Essential skills and experience:Proven data engineering experience with Databricks (Delta Lake, Unity Catalog, Apache Spark) in cloud-based data platforms and environments (AWS).Strong programming skills in Python and SQL, with experience ...

practices.What You'll DoArchitect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments.Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate.Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration.What We're Looking ForProven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments.Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms.Extensive experience ...

deliver actionable, enterprise-grade business insights.Key ResponsibilitiesData Architecture & CDL Governance: Design, implement, and govern the enterprise Common Data Layer (CDL) on Azure Databricks using Delta Lake and L0–L1–L2 architectures. Establish data quality frameworks, schema evolution standards, and deduplication logic.Pipeline & Integration Engineering: Oversee the creation of scalable … connection of Azure Databricks data assets into robust, secure, and intuitive Power BI dashboards.Required Experience & SkillsTechnical ExpertiseCloud Data Platforms: Advanced knowledge of Azure Databricks, Delta Lake, Azure Data Factory (ADF), and SQL querying.SAP Ecosystem: Deep data discovery skills within SAP S/4HANA, SAP BW/HANA ...

Medallion Architecture, and ensuring our Databricks platform is secure, governed, and production ready.Key Responsibilities: Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake.Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms).Implement strong data quality, lineage … across pipelines and clusters.Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets.Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required.Partner with analysts and data scientists to ensure that high-quality, well-modelled ...

including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices (testing ...

communicate complex concepts in a clear, concise manner• Languages: Terraform, Python, SQL, PowerShell.• CI/CD: Azure DevOps pipelines for data platform deployments.• Databricks: Delta Lake, Unity Catalog, Workflows, Notebooks, Clusters, MLflow.• Azure: ADLS Gen2, Azure Data Factory, Event Hubs/Kafka, Key Vault, Functions, Synapse. AI Foundry … Service Plan Deployment and Configuration, Integration with Azure bot Services• MS Fabric: Tenant Management, Workspace Private Links, Deployment, Capacity Management, Workspace Management, One Lake, Data Mirroring, Power BI• AI Foundry : Model management, Hosted Agents Deployment and Management , Project Configuration (Connections and Governance ) and lifecycle management.#LI-PM1Inclusion & DiversityWe value diversity ...

Strong Azure Data Factory (ADF) and Azure Databricks experience. Experience building ETL/ELT pipelines within Azure Lakehouse environments. Knowledge of Medallion Architecture and Delta Lake. Strong SQL and Python skills. Experience with Power BI datasets and data modelling. Familiarity with Azure DevOps, Git, CI/CD and Infrastructure ...

systematic approach to diagnosing and resolving issuesHighly desirable skillsExperience with streaming frameworks such as Flink, Kafka Streams and Kafka ConnectKnowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data CatalogExposure to DataOps practices and collaboration with Data Engineering teamsFamiliarity with GPU-accelerated analytics using ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

platforms (preferably AWS); production experience is not required.A collaborative mindset and willingness to learn and communicate openly within the team.Nice to haveExposure to data lake concepts or formats such as Delta Lake or Iceberg.Any experience with search, graph, or large-scale analytical databases.Interest in AI, machine learning ...

pipeline development, and large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security … Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most prominent corporations ...

files, and event streams Supporting master data management, data quality rules, and golden record definition Working hands-on with ADLS, Azure SQL, Event Hubs, Delta Lake, and OneLake Designing integration patterns and REST APIs to support near real-time data flows Contributing to infrastructure-as-code and environment ...

accelerate delivery, and enhance solution quality while applying sound engineering judgement and responsible AI practices.What You'll DoDesign and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services.Build real-time, event-driven solutions using technologies such as Kafka and Azure Event ...

interpersonal skills - you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environmentExperience in the financial markets, especially in delta one, store of value, and/or FICC options tradingExperience with Linux-based, concurrent, high-throughput, low-latency software systemsExperience with pipeline orchestration frameworks … e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about DRW's processing activities ...

improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration patterns for REST … Azure Databricks implementation and troubleshooting Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration ...

planning, solution design and adoption. Data Engineering and Integration Design, build and optimise scalable data ingestion and transformation solutions using Python, PySpark, SQL and Delta Lake. Implement batch and incremental processing patterns, including Change Data Capture (CDC), schema evolution, reconciliation processes, data quality controls and error handling. Develop reusable … administration and troubleshooting experience. Databricks Serverless architecture, workload placement and compute optimisation. Azure networking, identity, security, monitoring, secrets management and private connectivity. Databricks SQL, Delta Lake and query/workload performance optimisation. Strong understanding of platform governance, operational support and best practices. Highly Desirable Experience migrating workloads from ...

transform, and model people data from multiple HR and workforce systems — including Workday — into a governed, enterprise-grade data model built on Databricks and Delta Lake. Data Modelling & Standards: Own and evolve canonical people data structures across the global people data platform. Define and enforce schemas, ingestion patterns … data models in a cloud environment Strong proficiency in SQL; solid Python and/or PySpark capabilityDemonstrable experience with Databricks, Apache Spark, and Delta Lake architectureProven ability to design and own enterprise-grade canonical data models across multiple upstream source systemsExperience integrating data from APIs and diverse upstream ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectationsPreferred qualifications, capabilities, and skillsExperience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworksFamiliarity with AWS-native data services such as Glue, Lake Formation, Redshift, or AthenaExposure ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

workflows and automated testing. Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability. The Individual Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow. Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral. Experience building and deploying GenAI ...