26 to 50 of 58 Data Lake Jobs in London

Technology Product Owner, AI Data Analytics, Microsoft Stack, Azure, Data Engineering, ML, Azure, Remote with Travel Technology Product Owner required to join a global Professional Services business based in Central London. However, this is practically a remote role, but when travel is required (to London, Europe … Development FIRST mentality. You must have been a Developer before moving into Product Ownership Technical proficiency:Familiarity with Azure services (e.g., Data Lake, Synapse, Fabric) and Databricks for data engineering, analytics, performance optimisation, and governance. Experience with implementing and optimising scalable cloud infrastructure is highly ...

Protocol (MCP) for advanced context-aware analytics and automation. Develop and maintain security analytics and data pipelines within Sentinel Data Lake to support large-scale threat detection, incident response, and threat hunting, while optimizing cost and enabling Agentic AI-driven security operations. Integrate and automate … team of Cyber Defence Security Engineers. The Requirements: Must-Have Skills: Microsoft Sentinel Expertise : Architecture, deployment, management, including AI for Security, Data Lake optimization, MCP, and Sentinel Graph. Cloud Security Tools : Hands-on with Wiz Cloud (Defend, Runtime Sensor, Code), CSPM, and CWP. Automation & Integration : Using Sentinel ...

Data Engineering Consultant - £50,000 - Hybrid As a Data Engineer, you will work within an agile team to deliver high-value data solutions for clients. You'll be hands-on across the full lifecycle, using and developing your technical and project skills to create …/ELT pipelines using Azure Synapse, Data Factory, Databricks, or Fabric, with SQL and Python. Strong understanding of data lake and lakehouse architectures. Experience working with large, complex datasets from diverse sources. Strong SQL and Python capabilities, including stored procedures, notebooks, and query optimisation. Familiarity ...

Data Engineering Consultant - £50,000 - Hybrid As a Data Engineer, you will work within an agile team to deliver high-value data solutions for clients. You'll be hands-on across the full lifecycle, using and developing your technical and project skills to create …/ELT pipelines using Azure Synapse, Data Factory, Databricks, or Fabric, with SQL and Python. Strong understanding of data lake and lakehouse architectures. Experience working with large, complex datasets from diverse sources. Strong SQL and Python capabilities, including stored procedures, notebooks, and query optimisation. Familiarity ...

Data Engineering team lead - £70,000 - Hybrid Overview We're looking for a Lead Technical Consultant to guide a team of Technical Consultants, oversee delivery of high-quality data solutions, and collaborate across the business to ensure successful project outcomes. This role combines leadership, resource management … Innovation Support technical delivery when needed, including designing scalable Azure/Microsoft data solutions. Drive innovation through cloud migrations, data lake implementations, and robust ETL/ELT pipeline development. Skills & Experience You'll thrive in this role if you have: Leadership Proven ability to build ...

Data Engineering team lead - £70,000 - Hybrid Overview We're looking for a Lead Technical Consultant to guide a team of Technical Consultants, oversee delivery of high-quality data solutions, and collaborate across the business to ensure successful project outcomes. This role combines leadership, resource management … Innovation Support technical delivery when needed, including designing scalable Azure/Microsoft data solutions. Drive innovation through cloud migrations, data lake implementations, and robust ETL/ELT pipeline development. Skills & Experience You'll thrive in this role if you have: Leadership Proven ability to build ...

build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data. Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies. Conduct performance testing and cluster tuning to ensure cost-efficient, high-performing workloads. … governance best practices. Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles. Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management. Integrate Databricks solutions with Azure services such as Azure ...

build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data. Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies. Conduct performance testing and cluster tuning to ensure cost-efficient, high-performing workloads. … governance best practices. Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles. Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management. Integrate Databricks solutions with Azure services such as Azure ...

Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing. Youll design and build scalable data pipelines and maintain high-quality, test-driven code … pipelines for data ingestion, processing, and validation. Write unit and BDD tests using Behave, including mocking and patching. Create and optimize Delta Lake tables for reliable, performant data storage. Use Docker to manage consistent development, testing, and deployment environments. Build configurable, parameter-driven code ...

engineering solutions. Key requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with … Azure services: Data Lake Storage (Gen2), Key Vault, Azure Functions Familiarity with CI/CD for Databricks deployments Strong troubleshooting in distributed Data environments Excellent communication and collaboration skills Nice to have: Immediate availability Experience with enterprise-scale Databricks environments Knowledge of Azure Synapse, Power ...

engineering solutions. Key requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with … Azure services: Data Lake Storage (Gen2), Key Vault, Azure Functions Familiarity with CI/CD for Databricks deployments Strong troubleshooting in distributed Data environments Excellent communication and collaboration skills Nice to have: Immediate availability Experience with enterprise-scale Databricks environments Knowledge of Azure Synapse, Power ...

Python Data Engineer - Multi-Strategy Hedge Fund Location: London | Hybrid: 2 days per week on-site | Type: Full-time About the Role A leading multi-strategy hedge fund is seeking a highly skilled Python Data Engineer to join its technology and data team. This … scalable Python-based ETL pipelines for ingesting and transforming market data from multiple sources. Design and manage cloud-based data lake solutions (AWS, Databricks) for large volumes of structured and unstructured data. Implement rigorous data quality, validation, and cleansing routines to ensure accuracy ...

Senior Data Engineer Summer-Browning Associates is currently supporting our central government client, who is seeking a Senior Data Engineer for an initial three-month assignment with the option to extend. Location: Hybrid working - London, Bristol, Manchester, Birmingham, Newcastle, Exeter, Cambridge, Southampton The ideal candidate will … hold an active security clearance and strong experience in Data Engineering, with the following skills and qualifications: Proficiency in the Microsoft BI Stack, including Azure SQL, Fabric, Synapse Analytics, and Power BI. Experience in developing data products such as data warehouses, data ...

Data Engineer (Python) - Leading Asset Manager | London (Hybrid) Brand-new team. High autonomy. Big impact and high bonus! A top Asset Manager is building a new data function from scratch , and they need a sharp Data Engineer to help define and deliver their future … office, 2 days WFH (flexible when needed). The Role Build and optimise scalable data pipelines for a new Data Lake Work with Python, SQL, Spark, AWS, Docker, CI/CD Orchestrate workflows using Airflow, Prefect, or Dagster Support ETL, API integrations, and high-quality ...

Senior Data Engineering Consultant - £60,000 - Hybrid Key Responsibilities Lead, mentor, and develop a team of Technical Consultants. Manage resource planning, scheduling, and overall delivery workflows. Collaborate with Pre-sales, Commercial, and Project Management teams to scope … deliver projects. Contribute to technical delivery, designing scalable data solutions in Azure/Microsoft environments. Support cloud migrations, data lake builds, and ETL/ELT pipeline development. Ensure delivery follows best practices and internal standards. Skills & Experience Strong leadership and relationship-building skills. Experience guiding ...

Senior Data Engineering Consultant - £60,000 - Hybrid Key Responsibilities Lead, mentor, and develop a team of Technical Consultants. Manage resource planning, scheduling, and overall delivery workflows. Collaborate with Pre-sales, Commercial, and Project Management teams to scope … deliver projects. Contribute to technical delivery, designing scalable data solutions in Azure/Microsoft environments. Support cloud migrations, data lake builds, and ETL/ELT pipeline development. Ensure delivery follows best practices and internal standards. Skills & Experience Strong leadership and relationship-building skills. Experience guiding ...

reinsurer operating globally, with a focus on complex commercial and industrial risks. The business combines deep sector expertise with advanced use of data, analytics, and modern technology to deliver informed, efficient, and fair underwriting decisions. The company is in a period of significant growth and is committed … C# ecosystem and modern front-end frameworks (ideally Angular). Experience with data architecture concepts including relational, NoSQL, and data lake technologies. Knowledge of cloud-based AI architectures, including LLMs, Generative AI, and Agentic AI. Experience integrating proprietary systems with COTS finance or ERP platforms ...

reinsurer operating globally, with a focus on complex commercial and industrial risks. The business combines deep sector expertise with advanced use of data, analytics, and modern technology to deliver informed, efficient, and fair underwriting decisions.The company is in a period of significant growth and is committed to maintaining … C# ecosystem and modern front-end frameworks (ideally Angular). Experience with data architecture concepts including relational, NoSQL, and data lake technologies. Knowledge of cloud-based AI architectures, including LLMs, Generative AI, and Agentic AI. Experience integrating proprietary systems with COTS finance or ERP platforms ...

mining, and visualization to extract insights. Design and implement Big Data solutions using Hadoop, Spark, PySpark, and Data Lake architectures. Develop and deploy ML models (supervised, unsupervised, tree-based, ensemble methods). Implement deep learning architectures (CNN, RNN, LSTM) using TensorFlow, PyTorch … Analytical Tools: EDA, Data Mining, Visualization (Plotly, Matplotlib, Seaborn) Statistical & Multivariate Analysis Big Data: Hadoop, MapReduce, HDFS, DataBricks, Spark, PySpark DataLake Architecture, AWS Redshift, Kinesis, EMR Machine Learning Supervised Models: Nave Bayes, Logistic Regression, SVM, Linear Regression, KNN, Tree-Based Models: Decision Trees, Random Forest, Gradient ...

Data Platform Engineer - ADF, Terraform, Synapse, Spark £Market Rate (Inside IR35) London/Hybrid 6 months We are currently working with a client who urgently require a Data Platform Engineer with hands on expertise in Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing … Factory (ADF), including network configurations and integration runtime options. Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts. Experience in efficient handling of large-scale data Deep understanding of Azure DevOps pipelines and deploying Terraform (Infrastructure as Code ...

Data Platform Engineer - ADF, Terraform, Synapse, Spark £Market Rate (Inside IR35) London/Hybrid 6 months We are currently working with a client who urgently require a Data Platform Engineer with hands on expertise in Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing … Factory (ADF), including network configurations and integration runtime options. Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts. Experience in efficient handling of large-scale data Deep understanding of Azure DevOps pipelines and deploying Terraform (Infrastructure as Code ...

RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable … pipelines using Python and PySpark. Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies. Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads. Build and manage containerised environments using Docker for consistent development, testing ...

RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable … pipelines using Python and PySpark. Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies. Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads. Build and manage containerised environments using Docker for consistent development, testing ...

bonus, private healthcare and more! A large and growing retail business are looking to expand their IT function with the addition of a Data Integration Manager. This role will focus on end to end data engineering and integration solutions for a company wide modernisation programme. Duties … will include: - Managing the processes for data engineering and data integration in order to align to business needs - Leading a team across engineering and integration - Work to ensure the data infrastructure of the business effectively supports analytics, AL, machine learning ML and reporting - Designing ...

Data Engineer Remote - Azure - Databricks - up to £80k I'm working with a client who are a leading Microsoft consultancy that's growing fast and shaping the future of data. They are continuing to grow, driving innovation and help clients accelerate their cloud transformation journeys. With a pipeline … cutting-edge projects, this is your chance to work at the forefront of cloud data engineering and make a real impact. You'll Work With Designing and delivering scalable projects using Databricks, Synapse & Fabric Build and optimise ETL/ELT pipelines and data models with ...