1 to 25 of 106 Delta Lake Jobs in the UK

Data Factory, Microsoft Fabric, Fabric, SQL, PostgreSQL, API, CI/CD, PowerBI, ETL, ELT, Data Engineer, Azure fundamentals, Fabric Data Engineer, ADF, ADL, Data Lake, Delta Lake, Azure Data Warehouse, Azure Data Factory, Microsoft Fabric, Fabric, SQL, PostgreSQL, API, CI/CD, PowerBI, ETL, ELT, Data Engineer … Azure fundamentals, Fabric Data Engineer, ADF, ADL, Data Lake, Delta Lake, Azure Data Warehouse, Azure Data Factory, Microsoft Fabric, Fabric, SQL, PostgreSQL, API, CI/CD, PowerBI, ETL, ELT, Data Engineer, Azure fundamentals, Fabric Data Engineer, ADF, ADL, Data Lake, Delta Lake, Azure Data ...

Clearance: Active SC Clearance required Role Overview We are seeking an SC Cleared Python Data Engineer with strong hands-on experience in PySpark, Delta Lake, and Azure cloud services. The role focuses on designing and delivering scalable, well-tested data pipelines, with particular importance placed on the ability … both technical and non-technical stakeholders. Implement unit and BDD testing (Behave or similar), including effective mocking and dependency management. Design and optimise Delta Lake tables to support ACID transactions, schema evolution, and incremental processing. Build and manage Docker-based environments for development, testing, and deployment. Develop configuration ...

work involves developing and maintaining complex, test-driven data flows. You will be responsible for building scalable data pipelines using Python and PySpark, optimizing Delta Lake storage, and deploying solutions in a containerised Azure environment. Quality, maintainability, and automated testing are key to this role. What youll … flows with a strong focus on clean, testable Python code Writing unit and BDD tests using Behave, including mocking and patching Creating and optimizing Delta Lake tables for performance and reliability Working with Docker to support consistent development and deployment Integrating Azure services such as Azure Functions ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage. Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage . Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

Data Engineer/Data Engineering/Data Consultant/Lakehouse/Delta Lake/Data Warehousing/ETL/Azure/Azure Databricks/Python/SQL/Based in the West Midlands/Solihull/Birmingham area, Permanent role, £ + car/allowance (£5,000) + … allowance + bonus Experience: Experience in a Data Engineer/Data Engineering role Large and complex datasets Azure, Azure Databricks Microsoft SQL Server Lakehouse, Delta Lake Data Warehousing ETL Database Design Python/PySpark Azure Blob Storage Azure Data Factory Desirable: Exposure ML/Machine Learning/ ...

one. What You'll Be Doing Designing end-to-end Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. … impact projects with cutting-edge Microsoft technologies. What We're Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. ...

Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta Lake/Delta Live Tables and the Medallion architecture . Hands-on experience with AWS services such as S3, Glue, Lambda, Batch ...

analytics and data-driven decision making across the business. Key Responsibilities Design, develop, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. Build and optimise data lakehouse architectures on Azure Data Lake Storage ( ADLS ) . Develop high-performance data solutions using Azure Synapse, Azure … Azure DevOps Required Skills & Experience 3+ years' experience as a Data Engineer working in Azure environments. Strong hands-on experience with Databricks (PySpark, Delta Lake, cluster optimisation, job scheduling). Solid knowledge of Azure cloud services including: Azure Data Lake Storage Azure Data Factory Azure Synapse/ ...

optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with … Required Skills and Experience * Proven experience as a Data Engineer working with Azure cloud services. * Strong proficiency in Databricks, including PySpark, Spark SQL, notebooks, Delta Lake, and job orchestration. * Strong SQL and data modeling skills (e.g., dimensional modeling, data vault). * Experience with Azure Data Factory or other ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...