1 to 25 of 32 Permanent Delta Lake Jobs in England

Engineer - Leading UK Enterprise - Milton Keynes (Hybrid -2/3 times/month!) (Tech stack: Data Engineer, Python, Spark, SQL, Data Modelling, Data Architecture, Delta Lake, Cloud (Azure/AWS/GCP), Medallion Architecture) Our client is a leading UK enterprise undergoing a major data transformation programme … applicants should have strong experience in Python , Spark , and SQL , alongside solid expertise in Data Modelling and modern Data Architecture principles. Experience working with Delta Lake and Medallion Architecture is highly desirable. Exposure to Cloud (Azure, AWS or GCP) platforms is essential for any Data Engineer looking ...

pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED... Please either ...

data masking, encryption, lineage, access controls, and regulatory adherence (e.g., GDPR, ISO 27001). Design and optimize cloud-native storage and compute architectures using Delta Lake, ADLS Gen2, Synapse, Azure SQL, Cosmos DB, and NoSQL for petabyte-scale workloads. Implement event-driven and real-time streaming architectures using … culture. Your Profile Essential skills/knowledge/experience: Azure Data Factory, Synapse Analytics, Azure Databricks ADLS, Azure Blob, Azure SQL DB, Cosmos DB, Delta Lake, Oracle DB Azure Event Hub, Azure Kafka, Azure IoT Hub, ADX Fabric including OneLake CI/CD using Azure DevOps, GitHub ...

data masking, encryption, lineage, access controls, and regulatory adherence (e.g., GDPR, ISO 27001). Design and optimize cloud-native storage and compute architectures using Delta Lake, ADLS Gen2, Synapse, Azure SQL, Cosmos DB, and NoSQL for petabyte-scale workloads. Implement event-driven and real-time streaming architectures using … practices to promote a data-driven culture. Required Skills Azure Data Factory, Synapse Analytics, Azure Databricks ADLS, Azure Blob, Azure SQL DB, Cosmos DB, Delta Lake, Oracle DBA Azure Event Hub, Azure Kafka, Azure IoT Hub, ADX Fabric including OneLake CI/CD using Azure DevOps, GitHub ...

transformations, stored procedures, and performance basics 3+ years Azure Data Factory, including pipeline development, parameter usage, and basic templates 3+ years Databricks experience (PySpark, Delta Lake, notebooks) 2+ years Python Experience Experience working with data warehouses and structured modelling concepts Strong understanding of ETL/ELT practices … data centric engineering Desirable Exposure to metadata driven or config driven engineering practices Basic PySpark optimisation or Delta Lake performance tuning Experience using Azure DevOps (repos, pipelines) Understanding of data governance, data quality tooling, or lineage concepts Experience in regulated environments or financial services The Application Timeline ...

govern data products within a Databricks Lakehouse architecture. Key Responsibilities Design, develop, and optimise scalable ETL/ELT pipelines using Databricks, Python, SQL, and Delta Live Tables (DLT). Build and maintain performant Delta Lake solutions, including partitioning and optimisation strategies. Contribute to Lakehouse architecture design … commercial experience as a Data Engineer or Senior Data Engineer. Deep hands-on expertise with Databricks on AWS (essential). Strong understanding of Spark, Delta Lake, and Lakehouse architectures. Proven experience designing scalable and maintainable modern data platforms. Strong knowledge of the end-to-end data engineering lifecycle. ...

analytics tasks and queries. Exposure to Java script technologies (preferably React) Exposure to Azure Databricks environment is desirable (PySpark, notebooks, integrate repos, job workflows, delta lake, unity catalog etc) Exposure to azure environment and various interconnected azure components (Functions, App Service, Containers etc.). Knowledge on Financial domain … Bash, PySpark Web frameworks: Django, Flask, or FastAPI. Frontend Technologies: ReactJS, Java Script Azure : Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage Gen2, Azure Event Grid, Azure Event Hubs, Azure Service Bus, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure API Management (APIM), Azure DevOps. ...

Databricks , including target architecture, data models, security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration. Optimize … data engineers, platform teams, and business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability. Mandatory Skills: Advanced Databricks experience: architecture, Delta Lake, job orchestration, cluster management, performance tuning. Proven migration experience (Palantir → Databricks) necessary , including pipeline conversion, data model redesign, and production cutover. Expert ...

Doing Designing and building end-to-end data pipelines using Databricks on Azure or AWS Developing modern lakehouse architectures using: Databricks (Spark, SQL, Delta Lake) Azure Data Lake Storage or Amazon S3 Writing high-quality PySpark, Python, and SQL for ingestion, transformation, and modelling Implementing Bronze/ ...

tooling Key skills Strong experience with Microsoft Fabric (Data Factory, Lakehouse, OneLake, Pipelines, SQL Warehouse) Solid data engineering background using SQL, Python, PySpark, Delta Lake E xperience building and optimising end-to-end data pipelines CI/CD experience for analytics and data platforms (Azure DevOps or GitHub ...

structured/unstructured data Collaborate with engineering, business, and security teams Key Skills & Experience Azure Data Factory, Synapse, Databricks, Fabric/OneLake ADLS Gen2, Delta Lake, Cosmos DB, Azure SQL Kafka, Event Hub, IoT Hub, real-time streaming CI/CD, Terraform, Bicep, Azure DevOps, GitHub Data Governance ...

BigQuery, Redshift Experience in open source technologies like Spark, Kafka, Beam Good understanding of Cloud providers – AWS, Microsoft Azure, Google Cloud Familiarity with DBT, Delta Lake, Databricks Experience working in an agile environment Experience working with AI tools (Claude code, Cortex etc) Here’s a taste ...

such as Azure (especially services such as ADLS, Azure SQL Server, Key Vault) or similar platform services such as AWS/GCP ·Familiarity with Delta Lake and transactional data patterns ·Experience with data orchestration tools (e.g. Databricks Workflows, MS Scheduler) ·Understanding of data modelling concepts for analytics ...

target mapping, data quality rule setup for the target data model’s critical data elements (CDE). Database Optimization: Optimize Cloud databases such as Delta Lake or One Lake for efficiency, scalability, and faster access using techniques like indexing and partitioning. Data Governance: Work with Federated Governance ...

modern data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering practices (Airflow, dbt, Prefect) and governance (Collibra, Purview) Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP Lead MLOps strategy (MLflow, Kubeflow, SageMaker, Vertex ...

scale data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering and governance practices across a distributed global data estate Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) to support AI and analytics workloads Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP within ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines. Start Date: ASAPContract Type: Outside IR35Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner, we'd love ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

knowledge of Unity Catalog and Hive Metastore AWS IAM, S3, KMS, and Terraform experience Strong troubleshooting and integration support skills Desirable Python, PySpark, SQL Delta Lake/Iceberg CI/CD tools (GitLab, GitHub Actions, Azure DevOps) Data governance and lineage experience ...

PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake, PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data ...

Required Skills Strong background in data testing, ETL validation, and data migration. Hands-on expertise with cloud-based data platforms (Azure Data Factory, Databricks, Delta Lake). Advanced SQL skills for complex query validation. Experience with Agile/DevOps delivery models and JIRA test management. Familiarity with automation ...

multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience ...

experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory • Hands-on data engineering experience (Spark/PySpark, Delta Lake) • Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC • Experience leading Agile teams and managing risks, dependencies, and senior ...

data platform and services using the below technologies: Python, pandas and SQL in Jupyter notebooks for analysis dbt and Databricks data platform with Delta Lake Lightdash as our BI tool and semantic layer Streamlit for interactive data applications Airflow for orchestration Kubernetes for application deployment CI/ ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Strong communication skills and confidence presenting ideas and technical approaches. ...