51 to 75 of 476 Data Pipeline Jobs in London

Energy Domain is a must. Under 7 years of experience only. Permanent. Key Responsibilities Design, develop, and maintain data ingestion pipelines using open-source frameworks and tools Build and optimise ETL/ELT processes to handle small to … large-scale data processing requirements Develop data models and schemas that support analytics, business intelligence and product needs Monitor, troubleshoot, and optimise data pipeline performance and reliability Collaborate with stakeholders, analysts and product team to understand data requirements Implement data quality checks and validation processes to ensure data … Participate in architecture decisions and contribute to technical roadmap planning Technical Skills: Great SQL skills with experience in complex query optimization Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) Hands-on experience building and maintaining data ingestion pipelines Proven track record of optimising queries, code, and system performance Experience with More ❯

Energy Domain is a must. Under 7 years of experience only. Permanent. Key Responsibilities Design, develop, and maintain data ingestion pipelines using open-source frameworks and tools Build and optimise ETL/ELT processes to handle small to … large-scale data processing requirements Develop data models and schemas that support analytics, business intelligence and product needs Monitor, troubleshoot, and optimise data pipeline performance and reliability Collaborate with stakeholders, analysts and product team to understand data requirements Implement data quality checks and validation processes to ensure data … Participate in architecture decisions and contribute to technical roadmap planning Technical Skills: Great SQL skills with experience in complex query optimization Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) Hands-on experience building and maintaining data ingestion pipelines Proven track record of optimising queries, code, and system performance Experience with More ❯

Energy Domain is a must. Under 7 years of experience only. Permanent. Key Responsibilities Design, develop, and maintain data ingestion pipelines using open-source frameworks and tools Build and optimise ETL/ELT processes to handle small to … large-scale data processing requirements Develop data models and schemas that support analytics, business intelligence and product needs Monitor, troubleshoot, and optimise data pipeline performance and reliability Collaborate with stakeholders, analysts and product team to understand data requirements Implement data quality checks and validation processes to ensure data … Participate in architecture decisions and contribute to technical roadmap planning Technical Skills: Great SQL skills with experience in complex query optimization Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) Hands-on experience building and maintaining data ingestion pipelines Proven track record of optimising queries, code, and system performance Experience with More ❯

Energy Domain is a must. Under 7 years of experience only. Permanent. Key Responsibilities Design, develop, and maintain data ingestion pipelines using open-source frameworks and tools Build and optimise ETL/ELT processes to handle small to … large-scale data processing requirements Develop data models and schemas that support analytics, business intelligence and product needs Monitor, troubleshoot, and optimise data pipeline performance and reliability Collaborate with stakeholders, analysts and product team to understand data requirements Implement data quality checks and validation processes to ensure data … Participate in architecture decisions and contribute to technical roadmap planning Technical Skills: Great SQL skills with experience in complex query optimization Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) Hands-on experience building and maintaining data ingestion pipelines Proven track record of optimising queries, code, and system performance Experience with More ❯

more than just a media agency; we are people people . At our core lies a commitment to understanding, trust, integrity, and delivering results. With a focus on data-driven decision-making, we have built a reputation for driving tangible outcomes for our clients. As an employee-owned organisation, we value our team members' contributions and provide them … s future. Join us, and let's create meaningful change together. The Role As an AI Engineer, you will head up the design, build, and optimisation of our data infrastructure, automation solutions, and AI-driven tools. Leading Data Architecture, MarTech, AI & Automation, you will ensure that data flows seamlessly across the business and is … harnessed effectively to drive client performance. This role requires a mix of technical expertise, leadership, and client-facing skills. You will manage and develop team members, oversee data engineering projects, and take ownership of the agency's core data pipelines and platforms using Artificial Intelligence. You will also play a key role in exploring and embedding More ❯

more than just a media agency; we are people people . At our core lies a commitment to understanding, trust, integrity, and delivering results. With a focus on data-driven decision-making, we have built a reputation for driving tangible outcomes for our clients. As an employee-owned organisation, we value our team members' contributions and provide them … s future. Join us, and let's create meaningful change together. The Role As an AI Engineer, you will head up the design, build, and optimisation of our data infrastructure, automation solutions, and AI-driven tools. Leading Data Architecture, MarTech, AI & Automation, you will ensure that data flows seamlessly across the business and is … harnessed effectively to drive client performance. This role requires a mix of technical expertise, leadership, and client-facing skills. You will manage and develop team members, oversee data engineering projects, and take ownership of the agency's core data pipelines and platforms using Artificial Intelligence. You will also play a key role in exploring and embedding More ❯

more than just a media agency; we are people people . At our core lies a commitment to understanding, trust, integrity, and delivering results. With a focus on data-driven decision-making, we have built a reputation for driving tangible outcomes for our clients. As an employee-owned organisation, we value our team members' contributions and provide them … s future. Join us, and let’s create meaningful change together. The Role As an AI Engineer, you will head up the design, build, and optimisation of our data infrastructure, automation solutions, and AI-driven tools. Leading Data Architecture, MarTech, AI & Automation, you will ensure that data flows seamlessly across the business and is … harnessed effectively to drive client performance. This role requires a mix of technical expertise, leadership, and client-facing skills. You will manage and develop team members, oversee data engineering projects, and take ownership of the agency’s core data pipelines and platforms using Artificial Intelligence. You will also play a key role in exploring and embedding More ❯

are more than just a media agency; we are people people. At our core lies a commitment to understanding, trust, integrity, and delivering results. With a focus on data-driven decision-making, we have built a reputation for driving tangible outcomes for our clients. As an employee-owned organisation, we value our team members' contributions and provide them … s future. Join us, and let’s create meaningful change together. The Role As an AI Engineer, you will head up the design, build, and optimisation of our data infrastructure, automation solutions, and AI-driven tools. Leading Data Architecture, MarTech, AI & Automation, you will ensure that data flows seamlessly across the business and is … harnessed effectively to drive client performance. This role requires a mix of technical expertise, leadership, and client-facing skills. You will manage and develop team members, oversee data engineering projects, and take ownership of the agency’s core data pipelines and platforms using Artificial Intelligence. You will also play a key role in exploring and embedding More ❯

About the Opportunity This is a unique opportunity for a Data Engineer to work at the intersection of cutting-edge technology and elite sports performance. You’ll join a collaborative, high-standard engineering team with a culture built around creativity, innovation, and excellence. In this role, you’ll play a key part in designing and developing scalable data … football world—including performance insights for some of the most well-known footballers in the industry . This is the perfect environment for those passionate about leveraging advanced data engineering to drive real-world results in sports analytics. Key Responsibilities Architect, design, build, test, and maintain scalable and high-performance data pipelines Ensure all systems adhere … to best practices for data quality, integrity, and security Integrate modern data engineering tools and technologies into cloud-native infrastructure Develop tools and solutions to support sports data modeling, analytics, and predictive insights Collaborate with data scientists to enable end-to-end data workflows Own the architecture and maintenance of More ❯

About the Opportunity This is a unique opportunity for a Data Engineer to work at the intersection of cutting-edge technology and elite sports performance. You’ll join a collaborative, high-standard engineering team with a culture built around creativity, innovation, and excellence. In this role, you’ll play a key part in designing and developing scalable data … football world—including performance insights for some of the most well-known footballers in the industry . This is the perfect environment for those passionate about leveraging advanced data engineering to drive real-world results in sports analytics. Key Responsibilities Architect, design, build, test, and maintain scalable and high-performance data pipelines Ensure all systems adhere … to best practices for data quality, integrity, and security Integrate modern data engineering tools and technologies into cloud-native infrastructure Develop tools and solutions to support sports data modeling, analytics, and predictive insights Collaborate with data scientists to enable end-to-end data workflows Own the architecture and maintenance of More ❯

About the Opportunity This is a unique opportunity for a Data Engineer to work at the intersection of cutting-edge technology and elite sports performance. You’ll join a collaborative, high-standard engineering team with a culture built around creativity, innovation, and excellence. In this role, you’ll play a key part in designing and developing scalable data … football world—including performance insights for some of the most well-known footballers in the industry . This is the perfect environment for those passionate about leveraging advanced data engineering to drive real-world results in sports analytics. Key Responsibilities Architect, design, build, test, and maintain scalable and high-performance data pipelines Ensure all systems adhere … to best practices for data quality, integrity, and security Integrate modern data engineering tools and technologies into cloud-native infrastructure Develop tools and solutions to support sports data modeling, analytics, and predictive insights Collaborate with data scientists to enable end-to-end data workflows Own the architecture and maintenance of More ❯

Databricks Data Engineer £500 - £600 per day Outside IR35 We're partnering with a leading online retail company that's transforming the way data and real-time intelligence shape customer experiences. Their mission is to harness cutting-edge data and streaming technologies to drive smarter decisions, improve efficiency, and create personalised journeys for millions … of shoppers worldwide. The Role As a Senior Data Engineer, you'll play a key role in developing and optimising the backbone of the company's data platform. You'll be responsible for building and maintaining large-scale, real-time data pipelines that power analytics, machine learning, and operational systems across the business. You … ll collaborate with software engineers, data scientists, and analytics teams to ensure the platform delivers reliable, high-quality, and compliant data at scale. This is a hands-on engineering role that blends software craftsmanship with data architecture expertise. Key responsibilities: Design and implement high-throughput data streaming solutions using Kafka, Flink, or More ❯

Lead Data Engineer Hybrid (London/Reading - 1-2 days per week) Up to £90,000 + Benefits Are you ready to take the lead in shaping world-class data infrastructure at one of the most influential analytics companies on the planet? We're partnering with a global leader in marketing data and insights … that's seeking a Lead Data Engineer to help scale and evolve its cloud-based data ecosystem. This is an opportunity to work on high-impact global datasets , enabling better analytics, stronger fraud prevention, and smarter decision-making across 90+ markets. Why this role? Play a key role in building and optimising global data pipelines supporting over 50 consumer panels worldwide. Lead the technical strategy for data architecture, quality, and scalability. Collaborate closely with data scientists, analysts, and software engineers to drive innovation in data engineering. Hybrid flexibility: 1-2 days per week in London, Reading, or Southbank office , balance remotely. Join a global organisation with More ❯

Data Engineer - Azure, Databricks, Microsoft Fabric | Hybrid | £60,000 - £70,000 We're working with a leading financial services organisation that is investing heavily in its data transformation and analytics capabilities. They are looking for a Data Engineer to join their growing Data and Analytics team. This is ideal for someone with … a solid foundation in data engineering who wants to develop deeper skills in Azure Databricks and Microsoft Fabric . You will play a key role in developing and maintaining modern data pipelines, shaping the medallion architecture, and building high-quality data models that power reporting and advanced analytics across the business. What You'll … Do Build and maintain scalable data pipelines in Azure Databricks and Microsoft Fabric using PySpark and Python Support the medallion architecture (bronze, silver, gold layers) to ensure a clean separation of raw, refined, and curated data Design and implement dimensional models such as star schemas and slowly changing dimensions Work closely with analysts, governance, and engineering More ❯

Senior Data Engineer Rate: up to £450/day Outside IR35 Contract Length: 6 Months (with likely extension) Location: Hybrid - 1 day per week onsite in Central London Working hours: 9am - 5pm - Monday - Friday An established international organisation is seeking a Senior Data Engineer to join their growing Data & Middleware function, driving innovation and … excellence across complex data ecosystems. This is a fantastic opportunity to shape the future of a large-scale data transformation programme, working on high-impact projects that enable data-driven decision-making. Responsibilities for the Senior Data Engineer: Design, build, and maintain scalable data pipelines and architectures, ensuring reliability, performance … and best-in-class engineering standards Leverage Databricks, Spark, and modern cloud platforms (Azure/AWS) to deliver clean, high-quality data for analytics and operational insights Lead by example on engineering excellence, mentoring junior engineers and driving adoption of DevOps and CI/CD best practices within the data function Contribute to the evolution of More ❯

Part-Time Data Engineer/Power BI Developer (Fully Remote) 3-Month Contract | 3 Days per Week | Outside IR35 We’re looking for a hands-on Data Engineer/Power BI Developer to deliver a new Azure-based reporting solution for a London-based organisation. This is a part-time contract (3 days per week) and … can be completed fully remotely. The Project You’ll be building an end-to-end data reporting solution on Azure, pulling data from multiple third-party systems (initially Sona) into an Azure SQL environment, transforming it with Azure Data Factory, and surfacing insights through Power BI dashboards. This setup will evolve into a full … Looking For Proven experience with Azure SQL, Azure Data Factory, and Power BI (integration + dashboard design). Strong understanding of end-to-end data pipeline development. Experience automating and streamlining reporting workflows. Able to work independently and deliver remotely with minimal supervision. If you’re a self-sufficient Data Engineer/Power More ❯

Job Title: Staff Data Engineer Location: London, Hybrid Salary: c.£140,000 + bonus + share options Why Apply? This is a unique opportunity to take a leading role in shaping the data strategy of a fast growing Insurtech scale-up. The role combines hands on technical work with strategic influence, allowing you to develop both … systems and people. You'll lead a small but high impact team, work across multiple business functions, and help deliver a modern, scalable data platform that drives business decisions. Responsibilities: Lead, mentor, and develop a high performing team of Data Engineers, fostering a culture of learning and excellence. Define and execute the data engineering … roadmap aligned with business goals. Own and evolve the end-to-end data platform, including ingestion, transformation, governance, and real time processing. Design, build, and maintain scalable data pipelines, warehouses, and models to support product and analytics teams. Introduce and enforce best practices in testing, CI/CD, infrastructure as code, and observability. Write and review More ❯

Data QA Engineer – Remote-first - £55-65,000 Overview: As a Data QA Engineer, you will ensure the reliability, accuracy and performance of our client’s data solutions. Operating remotely, you will work closely with Data Engineers, Architects and Analysts to design and execute automated testing strategies that validate data pipelines, integrations and transformations across cloud platforms. Your work will underpin data quality and ensure that insights are built on trusted, consistent data. Key Responsibilities: Design and implement automated tests for data pipelines, ETL processes and APIs across AWS and Azure environments. Develop and maintain test frameworks that validate data quality, completeness … and integrity. Collaborate with engineering teams to identify data defects, root causes and performance issues. Support continuous integration and deployment by embedding automated testing into CI/CD pipelines. Validate source-to-target data mapping, transformations and business logic. Produce clear test documentation and communicate test results to technical and non-technical stakeholders. Monitor data More ❯

an AI Data Engineer to turn 3D assets and visuals into high-quality datasets for next-gen AI in digital play. Own the full data pipeline and help build scalable, creative systems at the crossroads of AI, data, and fun RESPONSIBILITIES Design and operate data ingestion pipelines for 3D assets, geometry … vendors, and automate quality checks to ensure scalable, reliable datasets. Define schemas and metadata for digital assets, maintaining catalogues, lineage, and versioning to ensure reproducibility and governance. Enforce data quality, drift detection, and compliance (GDPR/COPPA), including watermarking and traceability to safeguard IP and user privacy. Collaborate with AI infrastructure engineers to optimize data storage … and compute performance for large-scale processing. Work with AI engineers to define data specifications, support experimentation, and integrate active learning loops that improve AI models over time. REQUIREMENTS 2 years of proven experience in data engineering for ML/AI, with strong proficiency in Python, SQL, and distributed data processing (e.g., PySpark). More ❯

an AI Data Engineer to turn 3D assets and visuals into high-quality datasets for next-gen AI in digital play. Own the full data pipeline and help build scalable, creative systems at the crossroads of AI, data, and fun RESPONSIBILITIES Design and operate data ingestion pipelines for 3D assets, geometry … vendors, and automate quality checks to ensure scalable, reliable datasets. Define schemas and metadata for digital assets, maintaining catalogues, lineage, and versioning to ensure reproducibility and governance. Enforce data quality, drift detection, and compliance (GDPR/COPPA), including watermarking and traceability to safeguard IP and user privacy. Collaborate with AI infrastructure engineers to optimize data storage … and compute performance for large-scale processing. Work with AI engineers to define data specifications, support experimentation, and integrate active learning loops that improve AI models over time. REQUIREMENTS 2 years of proven experience in data engineering for ML/AI, with strong proficiency in Python, SQL, and distributed data processing (e.g., PySpark). More ❯

an AI Data Engineer to turn 3D assets and visuals into high-quality datasets for next-gen AI in digital play. Own the full data pipeline and help build scalable, creative systems at the crossroads of AI, data, and fun RESPONSIBILITIES Design and operate data ingestion pipelines for 3D assets, geometry … vendors, and automate quality checks to ensure scalable, reliable datasets. Define schemas and metadata for digital assets, maintaining catalogues, lineage, and versioning to ensure reproducibility and governance. Enforce data quality, drift detection, and compliance (GDPR/COPPA), including watermarking and traceability to safeguard IP and user privacy. Collaborate with AI infrastructure engineers to optimize data storage … and compute performance for large-scale processing. Work with AI engineers to define data specifications, support experimentation, and integrate active learning loops that improve AI models over time. REQUIREMENTS 2 years of proven experience in data engineering for ML/AI, with strong proficiency in Python, SQL, and distributed data processing (e.g., PySpark). More ❯

an AI Data Engineer to turn 3D assets and visuals into high-quality datasets for next-gen AI in digital play. Own the full data pipeline and help build scalable, creative systems at the crossroads of AI, data, and fun RESPONSIBILITIES Design and operate data ingestion pipelines for 3D assets, geometry … vendors, and automate quality checks to ensure scalable, reliable datasets. Define schemas and metadata for digital assets, maintaining catalogues, lineage, and versioning to ensure reproducibility and governance. Enforce data quality, drift detection, and compliance (GDPR/COPPA), including watermarking and traceability to safeguard IP and user privacy. Collaborate with AI infrastructure engineers to optimize data storage … and compute performance for large-scale processing. Work with AI engineers to define data specifications, support experimentation, and integrate active learning loops that improve AI models over time. REQUIREMENTS 2 years of proven experience in data engineering for ML/AI, with strong proficiency in Python, SQL, and distributed data processing (e.g., PySpark). More ❯

Data Engineer, Azure London/Hybrid £competitive + bonus + excellent benefits Azure, Fabric, Python Data Engineer with extensive cloud experience is sought by thriving financial services organisation in the City of London. As part of the central data platform team, you will be responsible for developing the Data Engineering standards to … be followed by federated data teams across the business. Within these standards our client is looking to create a Data Engineering development environment that has the flexibility to ensure engineers work as efficiently as possible while operating within the security constraints outlined by the business. Responsibilities Work with Product Owners to establish product requirements. Design and … develop observable and secure data pipelines. Source, integrate and clean data using medallion architecture. Ensure data completeness, integrity and validity. Create Lakehouses and Warehouses. Optimise data pipelines, processing workflows and Fabric warehouses for performance. Maintain Data Catalogue and ensure data duplication does not occur. Ensure data More ❯

Data Engineer £500 - £560 per day London - 1 day per week in office We're working with a leading global healthcare technology company who are building out their next-generation data platform, with a strong emphasis on automation, testing, and cloud-native engineering, and are looking for an experienced Data Engineer to join their … team. The Role You'll be part of a modern data engineering function that's implementing best-in-class data practices across ingestion, transformation, and orchestration layers. The environment is highly technical, collaborative, and fast-paced, giving you the opportunity to work across a wide variety of data sources and tools. Day-to-day … responsibilities include: Designing and developing DBT models and Airflow pipelines within a modern data stack. Building robust data ingestion pipelines across multiple sources - including external partners, internal platforms, and APIs. Implementing automated testing and CI/CD pipelines for data workflows. Performing data extraction and enrichment, including web scraping and parsing of More ❯

we are hiring for Technical Data Architect location: Central London Type : Permanent Hybrid role (2-3 days from client location) We are seeking a highly skilled TechnicalData Architect- with expertise in Databricks, PySpark, and modern data engineering practices. The ideal candidate will lead the design, development, and optimization of scalable data pipelines, while ensuring … data accuracy, consistency, and performance across the enterprise Lakehouse platform. This role requires strong leadership, technical depth, and the ability to collaborate with cross-functional teams. ________________________________________ Key Responsibilities Lead the design, development, and maintenance of scalable, high-performance data pipelines on Databricks. Architect and implement data ingestion, transformation, and integration workflows using PySpark, SQL … and Delta Lake. Guide the team in migrating legacy ETL processes to modern cloud-based data pipelines. Ensure data accuracy, schema consistency, row counts, and KPIs during migration and transformation. Collaborate with Data engineer , BI Engineers, and Security teams to define data standards, governance, and compliance. Optimize Spark jobs and Databricks clusters More ❯