11 of 11 Permanent Delta Lake Jobs in London

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage . Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

data engineering, data platforms and big data technologies Evaluate and improved existing data processes Essential Experiences: Advanced SQL skills Proficient in Databricks, including Spark, Delta Lake and MLflow Experience migrating SSIS to Databricks In depth understanding of ETL/ELT processes Knowledge of data warehousing concepts and architectures ...

that directly underpin operational decision-making and analytics for live energy assets. Key responsibilities Design, build, and maintain scalable data pipelines using Databricks (including Delta Live Tables). Develop robust ETL/ELT workflows ingesting data from operational, telemetry, and third-party systems. Optimise pipeline performance, reliability, and cost … from these platforms. Technical requirements Proven experience as a Data Engineer in production environments. Strong Python and SQL skills. Hands-on Databricks experience (DLT, Delta Lake; Unity Catalog desirable). Solid understanding of data modelling, data warehousing, and distributed systems. Experience with cloud data platforms (Azure preferred ...

post include; Develop, construct, test and maintain data architectures within large scale data processing systems. Develop and manage data pipelines using Azure Data Factory, Delta Lake and Spark. Utilise Azure Cloud architecture knowledge to design and implement scalable data solutions. Utilise Spark, SQL, Python, R, and other data ...

responding to business and project needs rather than operating as a narrow silo. You'll work hands-on with Azure Databricks, Azure Data Factory, Delta Lake, and Power BI to create scalable data models, automated pipelines, and self-service analytics capabilities. This is a fantastic opportunity to join ...

trading workflows. Exposure to cloud platforms (Azure, AWS, or GCP). Experience hiring, mentoring, or leading small engineering teams. Familiarity with Apache Spark, Kafka, Delta Lake, or similar data technologies. Apply Now to avoid disappointment! To find out more about Huxley, please visit ...

consumption layers Technology Leadership Evaluate and recommend modern data technologies and tooling Stay current with industry trends such as open table formats (Apache Iceberg, Delta Lake), data observability, and cloud-native services Lead proofs-of-concept and technical assessments for new technologies Ensure the data architecture remains modern ...

Role and Responsibility The role focuses on designing, building and maintaining scalable data pipelines and products on a modern Databricks Lakehouse environment, using Delta Lake, Spark, Python (PySpark) and SQL to deliver high-quality, well-governed data across the business. You'll manage and optimise the cloud-based ...