1 to 25 of 43 Delta Lake Jobs in the UK

Engineer - Leading UK Enterprise - Milton Keynes (Hybrid -2/3 times/month!) (Tech stack: Data Engineer, Python, Spark, SQL, Data Modelling, Data Architecture, Delta Lake, Cloud (Azure/AWS/GCP), Medallion Architecture) Our client is a leading UK enterprise undergoing a major data transformation programme … applicants should have strong experience in Python , Spark , and SQL , alongside solid expertise in Data Modelling and modern Data Architecture principles. Experience working with Delta Lake and Medallion Architecture is highly desirable. Exposure to Cloud (Azure, AWS or GCP) platforms is essential for any Data Engineer looking ...

pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED... Please either ...

decisions, and working with people who care about building things properly, you’ll fit right in. You’ll Work With Azure Data Factory, Data Lake, Key Vault, Azure Functions Databricks, Delta Lake, Unity Catalog, PySpark, Spark SQL Medallion architecture, Lakehouse patterns, distributed compute CI/CD pipelines … home You’ll thrive in this role if you have: Strong hands‐on experience with Azure Data Services and Databricks Solid understanding of Delta Lake, Medallion architecture and Lakehouse patterns Proficiency in Python, PySpark and Spark SQL Experience building CI/CD pipelines using Azure DevOps Knowledge ...

access. Ensure backend applications are secure, scalable, and high-performing. Databricks & Data Engineering Build ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize data pipelines and queries for performance and cost efficiency in Azure or AWS cloud environments. … hands-on experience in Python development. Proficiency in React.js and modern frontend development. Strong knowledge of SQL and relational databases. Experience with Databricks, PySpark, Delta Lake, and Unity Catalog. Familiarity with Azure or AWS cloud environments. Understanding of REST APIs, CI/CD, and Git-based workflows. Preferred ...

govern data products within a Databricks Lakehouse architecture. Key Responsibilities Design, develop, and optimise scalable ETL/ELT pipelines using Databricks, Python, SQL, and Delta Live Tables (DLT). Build and maintain performant Delta Lake solutions, including partitioning and optimisation strategies. Contribute to Lakehouse architecture design … commercial experience as a Data Engineer or Senior Data Engineer. Deep hands-on expertise with Databricks on AWS (essential). Strong understanding of Spark, Delta Lake, and Lakehouse architectures. Proven experience designing scalable and maintainable modern data platforms. Strong knowledge of the end-to-end data engineering lifecycle. ...

analytics tasks and queries. Exposure to Java script technologies (preferably React) Exposure to Azure Databricks environment is desirable (PySpark, notebooks, integrate repos, job workflows, delta lake, unity catalog etc) Exposure to azure environment and various interconnected azure components (Functions, App Service, Containers etc.). Knowledge on Financial domain … Bash, PySpark Web frameworks: Django, Flask, or FastAPI. Frontend Technologies: ReactJS, Java Script Azure : Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage Gen2, Azure Event Grid, Azure Event Hubs, Azure Service Bus, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure API Management (APIM), Azure DevOps. ...

Data Engineer will design, build, support and operate data pipelines and core data platform technologies within the organisation's Azure-based analytics and lake house environment. The role is hands-on and implementation-focused, working within established architectural patterns to deliver reliable, secure and well-governed data for analytics … Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including performance monitoring and configuration Build Gold-layer datasets optimised for Power ...

framework for RC, PFE and EAD calculations, including netting sets, supervisory factors and collateral/CSA logic.* **Design a modern Databricks architecture** using Delta Lake, Unity Catalog and Spark/PySpark to support versioned, reproducible and regulator‐defensible exposure calculations.* **Develop high‐performance compute pipelines** that support intraday … Deep understanding of SA‐CCR** (RC, PFE, EAD), exposure methodology, netting sets, supervisory parameters and collateral/CSA design.* **Strong Databricks engineering capability**, including Delta Lake versioning, Unity Catalog governance, Spark/PySpark optimisation and secure data architecture.* **Experience designing governed, audit‐ready systems**, with a focus ...

reliability of existing workloads. Key Responsibilities Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL Develop and manage Delta Lake-based data models, supporting a structured Lakehouse/Medallion architecture (Bronze, Silver, Gold layers) Support the ingestion and transformation of large-scale data … Strong commercial experience working with Azure Databricks as a core data processing platform Deep expertise in: PySpark/Apache Spark for distributed data processing Delta Lake (ACID transactions, optimisation, data versioning) Spark SQL and advanced SQL techniques Python for data engineering and pipeline development Hands-on experience designing ...

reliability of existing workloads. Key Responsibilities Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL Develop and manage Delta Lake-based data models , supporting a structured Lakehouse/Medallion architecture (Bronze, Silver, Gold layers) Support the ingestion and transformation of large-scale data … Strong commercial experience working with Azure Databricks as a core data processing platform Deep expertise in: PySpark/Apache Spark for distributed data processing Delta Lake (ACID transactions, optimisation, data versioning) Spark SQL and advanced SQL techniques Python for data engineering and pipeline development Hands-on experience designing ...

Doing Designing and building end-to-end data pipelines using Databricks on Azure or AWS Developing modern lakehouse architectures using: Databricks (Spark, SQL, Delta Lake) Azure Data Lake Storage or Amazon S3 Writing high-quality PySpark, Python, and SQL for ingestion, transformation, and modelling Implementing Bronze/ ...

DevOps, Power BI, Fabric workspaces AWS cloud, S3, cross-cloud connectivity, API Gateways GCP (BigQuery, GCS, Dataflow) Deep expertise in: Databricks (Lakehouse architecture, Delta Lake, Unity Catalog, Delta Sharing, Pyspark, Spark SQL etc.) Snowflake (data warehousing & data sharing, Iceberg, ) Strong understanding of: Data architecture patterns, industry benchmarking ...

pipelines using PySpark and SQL Design and implement ETL/ELT processes for batch and streaming data Develop data solutions using Databricks Lakehouse and Delta Lake Ingest and integrate data from internal and external sources (e.g. Kafka, CDC) Optimise Spark jobs and data workflows for performance, scalability … build scalable data solutions Strong proficiency in Python and SQL Hands-on experience with Apache Spark (including Structured Streaming) Experience with Databricks (Workflows, Delta Live Tables, Lakehouse architecture) Experience with cloud platforms (AWS, Azure, or GCP) Experience with Terraform or similar infrastructure-as-code tools Experience working with structured ...

DevOps, Power BI, Fabric workspaces AWS cloud, S3, cross-cloud connectivity, API Gateways GCP (BigQuery, GCS, Dataflow) Deep expertise in: Databricks (Lakehouse architecture, Delta Lake, Unity Catalog, Delta Sharing, Pyspark, Spark SQL etc.) Snowflake (data warehousing & data sharing, Iceberg, ) Strong understanding of: Data architecture patterns, industry benchmarking ...

including technical leadership responsibilities Strong SQL skills and extensive experience building ETL/ELT pipelines at scale Hands-on expertise with Databricks, including Autoloader, Delta Live Tables, Delta Lake, and Unity Catalog Strong experience designing and implementing lakehouse and medallion architecture patterns Strong data modelling experience, including ...

tooling Key skills Strong experience with Microsoft Fabric (Data Factory, Lakehouse, OneLake, Pipelines, SQL Warehouse) Solid data engineering background using SQL, Python, PySpark, Delta Lake E xperience building and optimising end-to-end data pipelines CI/CD experience for analytics and data platforms (Azure DevOps or GitHub ...

exposure to mParticle, Tealium, Segment or Salesforce CDP will translate. Production-grade PySpark/Databricks experience — building scalable data pipelines, optimising performance, and operating Delta Lake/Lakehouse architectures in a real production environment. Solid DevOps craft — CI/CD pipelines, Git-based workflows, infrastructure-as-code (Terraform ...

including technical leadership responsibilities • Strong SQL skills and extensive experience building ETL/ELT pipelines at scale • Hands-on expertise with Databricks, including Autoloader, Delta Live Tables, Delta Lake, and Unity Catalog • Strong experience designing and implementing lakehouse and medallion architecture patterns • Strong data modelling experience, including ...

modern data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering practices (Airflow, dbt, Prefect) and governance (Collibra, Purview) Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP Lead MLOps strategy (MLflow, Kubeflow, SageMaker, Vertex ...

such as Azure (especially services such as ADLS, Azure SQL Server, Key Vault) or similar platform services such as AWS/GCP ·Familiarity with Delta Lake and transactional data patterns ·Experience with data orchestration tools (e.g. Databricks Workflows, MS Scheduler) ·Understanding of data modelling concepts for analytics ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

scale data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering and governance practices across a distributed global data estate Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) to support AI and analytics workloads Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP within ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines. Start Date: ASAPContract Type: Outside IR35Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner, we'd love ...