1 to 25 of 490 Data Pipeline Jobs in the UK

Engineer to join our Service Operations team. This role offers an exciting opportunity to learn, contribute to, and grow within the critical areas of data pipeline development, maintenance, and optimisation for our client-facing products and services. Working closely with experienced developers, data analysts, data scientists, and other stakeholders, you will … be instrumental in ensuring the efficient and reliable flow of data, building foundational skills in a supportive environment. Key Responsibilities Data Pipeline Development • Contribute to the design, building, and maintenance of scalable data pipelines • Implement ETL (Extract, Transform, Load) processes to ingest data from various sources • Ensure data quality … for data access and manipulation Documentation and Reporting: • Document data processes, pipelines, and architectures • Generate reports and visualisations to communicate data insights and pipeline performance Technical Support: • Provide technical support for data-related issues and troubleshoot problems as they arise • Assist in the implementation of data governance and compliance More ❯

The Opportunity Are you passionate about data engineering? The Open Data team at the Parliamentary Digital Service (PDS) is beginning the development of a new Open Data Platform for parliamentary procedure data, replacing existing platforms and their associated data integration and publishing services. As our Senior Data Engineer … you will play a key role in designing, building, and evolving the data platforms that support decision-making across the organisation. This role sits at the intersection of engineering, analytics, and product delivery. You'll work closely with cross-functional teams to provide high-quality, reliable, and scalable data solutions that drive insight and operational excellence. … experienced Data Engineer who will play a central role in building and supporting the new platform. The role will focus on developing the ETL data pipeline framework to integrate upstream master data from parliamentary business systems, as well as engineering new data services, including APIs and web-client applications. The Data More ❯

About The Role Job level: 10 Were looking for a Lead Data Engineer to join our Data Engineering and Analytics practice. In this role, you will: Lead the design, development, management and optimisation of data pipelines to ensure efficient data flows, recognising and sharing opportunities to reuse data flows where … possible. Coordinate teams and set best practices and standards when it comes to data engineering principles. Champion data engineering across projects and clients. Responsibilities Lead by example, holding responsibilities for team culture, and how projects deliver the most impact and value to our clients. Be accountable for the strategic direction, delivery and growth of our work. … understanding of data wareho using concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical and non More ❯

Head of Data Engineering - Preston A Head of Data Engineering/Data Engineering Manager to lead the design, development, and enhancement of the data infrastructure and pipelines is required by a leading company based in Preston. The role offers Hybrid working, so 2 - 3 days in the office a week. You must … solutions for delivering high-quality care. The ideal candidate will lead the design and execution of cloud-based based scalable data storage solutions, oversee ETL pipeline development and optimisation, establish and manage data schemas and dictionaries, develop data integration solutions, and lead data cleansing, validation, and enrichment processes. Duties … You'll be on the lookout for performance bottlenecks and working to optimise data processing and query performance. This will help make sure your data pipeline is scalable and ready to handle big loads. You'll also be setting up monitoring frameworks to keep an eye on how your data pipeline is More ❯

Data Pipeline Engineer (On prem to AWS migration)UtilitiesPredominantly remote: occasional travel to Bristol6 months+£700 - £750 per day In short: We require 3 Data Engineers to join a large on prem to AWS migration programme in replicating data pipelines in readiness for a lift and shift to the cloud. Essential: AWS and … Redshift In full: Role purpose Reporting to the Lead Data Engineer, the Data Engineer is responsible for designing and maintaining scalable data pipelines, ensuring data availability, quality, and performance to support analytics and operational decision-making. They contribute to the data engineering roadmap through research, technical vision, business alignment, and … Ensure quality is maintained throughout the data engineering lifecycle, delivering robust, scalable, and cost-effective solutions on time and within budget. Monitor and drive data pipeline lifecycle, from design and implementation through to optimisation and post-deployment performance analysis. Support operational resilience, ensuring data solutions are maintainable, supportable, and aligned with architectural principles. More ❯

Lead Data Engineer - Preston A Lead Data Engineer to lead the design, development, and enhancement of the data infrastructure and pipelines is required by a leading company based in Preston. The role offers Hybrid working, so 2 - 3 days in the office a week. You must have the following: Proven experience as a Senior … solutions for delivering high-quality care. The ideal candidate will lead the design and execution of cloud-based based scalable data storage solutions, oversee ETL pipeline development and optimisation, establish and manage data schemes and dictionaries, develop data integration solutions, lead data cleansing, validation, and enrichment processes. Duties include … You'll be on the lookout for performance bottlenecks and working to optimise data processing and query performance. This will help make sure your data pipeline is scalable and ready to handle big loads. Also be setting up monitoring frameworks to keep an eye on how your data pipeline is doing. Be More ❯

global open economy that works for everyone. We exist behind the scenes, securely connecting companies - from growth to enterprise - to thousands of banks worldwide, enabling them to access data and initiate payments through the power of open banking. What we're looking for As a Java Software Engineer specializing in Data Products at Yapily, you will … Database Management: Background in managing both SQL and NoSQL databases. Version Control & CI/CD: Knowledge of version control (Git) and CI/CD practices for data pipeline deployment and Exposure to tools such as Terraform. Data Modelling & Schema Design: Familiarity with data modelling and schema design for operational or analytical systems. API … compliance principles (GDPR, ISO27001). SaaS/API-driven Environments: Background in SaaS/API-driven environments, ideally with experience in billing or usage-based data. Data Pipeline Monitoring & Troubleshooting: Basic skills in monitoring and troubleshooting data pipelines. Data Security: Understanding of data security best practices and encryption for sensitive data. More ❯

support EQ's transition into a stronger, more profitable, product-led business, driven by real insights and the needs of its customers. We are looking for a skilled Data Engineer to join our Data team. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data pipelines … using Microsoft Fabric and Databricks. You will work closely with our Product and Engineering teams to ensure data flow and integration across our data ecosystem. Your expertise will help us to improve our data infrastructure, improve data quality, and enable data-driven decision-making across the organization. Core Duties and … experience in data engineering, with a focus on cloud-based data pipelines and architectures Strong expertise in Microsoft Fabric and Databricks, including data pipeline development, data warehousing, and data lake management Proficiency in Python, SQL, Scala, or Java Experience with data processing frameworks such as Apache Spark More ❯

IR35) Databricks Engineer needed with active SC Security Clearance for 6 Month Contract based in Central London (Hybrid). Developing a cutting-edge Azure Databricks platform for economic data modelling, analysis, and forecasting. Start ASAP in Dec 2025. Hybrid Working - 2 days/week remote (WFH), and 3 days/week working on-site from the Central London … office. A chance to work with a leading global IT and Digital transformation business specialising in Government projects: In-depth Data Engineering + strong hands-on Azure Databricks expertise. Azure Data Services, Azure Data Factory, Azure Blob Storage + Azure SQL Database. Designing, developing, building + optimising data pipelines, implementing data … including data modelling techniques + data integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data More ❯

IR35) Databricks Engineer needed with active SC Security Clearance for 6 Month Contract based in Central London (Hybrid). Developing a cutting-edge Azure Databricks platform for economic data modelling, analysis, and forecasting. Start ASAP in Dec 2025. Hybrid Working - 2 days/week remote (WFH), and 3 days/week working on-site from the Central London … office. A chance to work with a leading global IT and Digital transformation business specialising in Government projects: In-depth Data Engineering + strong hands-on Azure Databricks expertise. Azure Data Services, Azure Data Factory, Azure Blob Storage + Azure SQL Database. Designing, developing, building + optimising data pipelines, implementing data … including data modelling techniques + data integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data More ❯

Position: Analytics Engineer Location: London (hybrid 3 times a week)Department: ITType: 3 months rolling contractOutside IR35 The client is seeking an experienced Analytics Engineer to join its Data & Analytics team. The role focuses on building scalable data pipelines, transforming raw data into clean, reusable datasets, and enabling data-driven decision-making. … Key Responsibilities Design and build data products, with proficiency throughout the data lifecycle. Develop robust data models through close collaboration with business users and the engineering team. Partner with senior management, operational leads, and other stakeholders, coaching and supporting a data-driven culture, including KPI definition and reporting frameworks. Accountable for data extraction, transforming JSON & XML, utlising high experience within metadata management. Collaborate with data engineers to develop data, enrich product design and integrate data for predictive models or machine learning. Deliver well-defined, transformed, tested, documented, and code-reviewed datasets for analysis. Evaluate and recommend improvements in data flow, influencing and More ❯

Engineer Location: London (hybrid 3 times a week) Department: IT Type: 3 months rolling contract Outside IR35 The client is seeking an experienced Analytics Engineer to join its Data & Analytics team. The role focuses on building scalable data pipelines, transforming raw data into clean, reusable datasets, and enabling data-driven decision-making. … Key Responsibilities Design and build data products, with proficiency throughout the data lifecycle. Develop robust data models through close collaboration with business users and the engineering team. Partner with senior management, operational leads, and other stakeholders, coaching and supporting a data-driven culture, including KPI definition and reporting frameworks. Accountable for data extraction, transforming JSON & XML, utlising high experience within metadata management. Collaborate with data engineers to develop data, enrich product design and integrate data for predictive models or machine learning. Deliver well-defined, transformed, tested, documented, and code-reviewed datasets for analysis. Evaluate and recommend improvements in data flow, influencing and More ❯

DATA ARCHITECT - DV CLEARED NEW PERMANENT JOB OPPORTUNITY AVAILABLE WITHIN A LEADING NATIONAL SECURITY SME FOR A DATA ARCHITECT WITH ENHANCED DV CLEARANCE Permanent job opportunity for a Data Architect Leading National Security SME Salary up to £100,000 + Bonus London based organisation in an easily accessible location To apply please call or … email WHO WE ARE? We are recruiting multiple a Data Architect to support urgent National Security & Defence projects in London. Due to the nature of these projects you must hold enhanced DV Security Clearance. WHAT WILL THE DATA ARCHITECT BE DOING? You will be joining a leading SME who is working hard to support National Security … projects within UK Govt. Departments in London. As part of a team, you will be responsible for designing and implementing Data Solutions in Mission-Critical areas. WE NEED THE DATA ARCHITECT TO HAVE.... Current DV clearance - Enhanced Good at understanding complexity and abstracting that into a form that is consumable for a non-technical audience. Experience More ❯

Azure/Databricks Data Engineer x2 - UK Wide (Hybrid Working) £72,000-£80,000 Duties As a Databricks Data Engineer with an Azure focus, you will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design … develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Skills/Experience Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. … AI can support you in your delivery work Solid experience as a Data Engineer or similar role. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data More ❯

If you are a natural collaborator and a creative problem solver - this position might be for you! We are looking for a Senior Data Engineer to join our Data team and be a part of Nest's transformation to use its data and insight to drive decision making.You’ll work closely with stakeholders to … develop, test, and maintain data products and solutions that align with their requirements. You’ll be helping to develop and maintain a cloud-based data and analytics platform on Azure and help end-users of every level integrate their workflows between this and other platforms, including AWS-based solutions.You’ll combine data from different … sources, and build data pipelines to transform and organise raw data into formats and structures that can be easily used for business intelligence, visualisations and dashboards, deep dive analysis, and long term forecasting/modelling. To be successful in this positions you will have the following skills, competencies and experience: You’re a natural collaborator and More ❯

what we do that matters–it's how we do it. DRW is a place of high expectations, integrity, innovation and a willingness to challenge consensus. As a Data Engineer on our Data Experience team, you will play an integral role in bringing vendor datasets into our data platform, governing our centralized data pipelines, supporting rapid data product development, and working alongside individual Traders, Quantitative Researchers, and Back-Office personnel to best utilize the firm's data and platform tools. Technical Requirements Summary: Have experience designing and building data pipelines Have experience working within modern batch or streaming data ecosystems An expert in … their associated trade-offs. Experience with multiple data formats and serialization systems (e.g. Arrow, Parquet, Protobuf/gRPC, Avro, Thrift, JSON, etc.) Experience managing data pipeline orchestration systems (e.g. Kubernetes, Argo Workflows, Airflow, Prefect, Dagster, etc.) Proven experience in managing the operational aspects of large data pipelines such as backfilling datasets, rerunning batch More ❯

Engineer, you will operate within an Agile delivery environment, working closely with the Data Product Manager and Data Architect to ensure your team maintains a pipeline of delivery against the Backlog; providing vital insight from our wide-ranging dataset to support executive and operational decision making that will underpin sustained growth of BFS business units … solutions. You will also have management responsibilities for a small team of Data Engineers, who you will coach, support and organise to ensure we sustain a predictable pipeline of delivery, whilst ensuring all appropriate governance and best practice is adhered to. Your responsibilities as our Senior Databricks Engineer will include: Understand the business/product strategy and … Azure DevOps, collaborating on Sprint cycles, code deployment, version control, and development practices Develop and maintain data models, schemas, and documentation Monitor and troubleshoot data pipeline issues, ensuring timely resolution Stay up-to-date with the latest industry trends and technologies in data engineering and recommend improvements to existing systems or processes as More ❯

Greetings, We are hiring Data Archietct Position in London, UK London 3 days/week 6 Months contract JD is below: Responsible for designing and implementing Aviva target data architecture, working closely with business stakeholders, data analysts, and data engineers to understand the data requirements and objectives, and to design … data models, schemas, and policies that meet those needs. The role holder will have the following technical data skills: Ability to use data modelling techniques such as entity-relationship diagrams, dimensional modelling, and schema design to represent data structures and relationships in a logical and conceptual way. data governance tools … like data catalogues, dictionaries, lineages, and policies in order to define and enforce the rules for managing the data lifecycle and usage. Proficiency in the use of database management systems like SQL, NoSQL, or cloud-based databases to store, query, and manipulate data in a structured and consistent way; o Data integration More ❯

Data Scientist Data Scientist Initial 6-month contract £475 to £500 per day Outside IR35 Remote Working (+ occasional site visits) The Opportunity ISR are supporting a major UK Government Agency seeking an experienced Data Scientist to help shape and expand their data and analytics capabilities. You’ll play a key role … in designing and delivering advanced data models, integrating new sources of information and applying modern AI and machine learning techniques to solve real-world challenges. This role is ideal for someone who thrives at the intersection of data science, analytics, and public sector transformation, helping to deliver better outcomes for the general public through data … clustering algorithms, text embeddings, LangChain and LLM deployment. Strong knowledge of data visualisation tools such as Power BI, DAX and Power Query. Experience in data pipeline automation and data governance in large-scale environments (Desirable). Familiarity with modern AI frameworks and open-source toolkits (Desirable). Role and Responsibilities Design and implement More ❯

COSTAR GROUP - SENIOR DATA ENGINEER ONTHEMARKET TECHNOLOGY - LONDON OVERVIEW CoStar Group (NASDAQ: CSGP) is a leading global provider of commercial and residential real estate information, analytics, and online marketplaces. Included in the S&P 500 Index and the NASDAQ 100, CoStar Group is on a mission to digitize the worlds real estate, empowering all people to discover properties … content, imagery, design and functionality. The OnTheMarket product will support home buyers, sellers, and agents better than ever! Learn more about OnTheMarket . ROLE DESCRIPTION The role ofSenior Data Engineerwill be a key role within the OTM Data team, reporting to the Head of Data and working closely with the CTO. The Data team is responsible for data acquisition, data loading, data quality, data processing, data delivery, and data support for OnTheMarket. This role will provide focus on delivering value to our customers, by leveraging data to improve the OTM users and agents experience. This role would More ❯

An opportunity has arisen for a Senior Data Engineer to join a well-established biotech company using large-scale genetic data and AI to predict disease risk and advance precision healthcare. As a Senior Data Engineer , you will be responsible for developing, automating, and optimising scalable data pipelines using modern cloud technologies. … with hybrid/remote working options offering a salary of £500 - £650 per day (Inside IR35) and benefits. You Will Be Responsible For: Designing and implementing cloud-based data architectures using Azure services. Building robust and scalable data pipelines to support complex, high-volume processing. Deploying and managing containerised workloads through Kubernetes, Helm, and Docker. Automating … Engineer, Data Engineer, Data Platform Engineer, Data Architect, Data Infrastructure Engineer, Cloud Data Engineer, DataOps Engineer, Data Pipeline Engineer, Devops Engineer or in a similar role. Proven experience with Azure cloud platforms and related architecture. Highly skilled in Python for data engineering, scripting, and automation. More ❯

An opportunity has arisen for a Senior Data Engineer to join a well-established biotech company using large-scale genetic data and AI to predict disease risk and advance precision healthcare. As a Senior Data Engineer , you will be responsible for developing, automating, and optimising scalable data pipelines using modern cloud technologies. … with hybrid/remote working options offering a salary of £500 - £650 per day (Inside IR35) and benefits. You Will Be Responsible For: Designing and implementing cloud-based data architectures using Azure services. Building robust and scalable data pipelines to support complex, high-volume processing. Deploying and managing containerised workloads through Kubernetes, Helm, and Docker. Automating … Engineer, Data Engineer, Data Platform Engineer, Data Architect, Data Infrastructure Engineer, Cloud Data Engineer, DataOps Engineer, Data Pipeline Engineer, Devops Engineer or in a similar role. Proven experience with Azure cloud platforms and related architecture. Highly skilled in Python for data engineering, scripting, and automation. More ❯

An opportunity has arisen for a Senior Data Engineer to join a well-established biotech company using large-scale genetic data and AI to predict disease risk and advance precision healthcare. As a Senior Data Engineer , you will be responsible for developing, automating, and optimising scalable data pipelines using modern cloud technologies. … with hybrid/remote working options offering a salary of £500 - £650 per day (Inside IR35) and benefits. You Will Be Responsible For: Designing and implementing cloud-based data architectures using Azure services. Building robust and scalable data pipelines to support complex, high-volume processing. Deploying and managing containerised workloads through Kubernetes, Helm, and Docker. Automating … Engineer, Data Engineer, Data Platform Engineer, Data Architect, Data Infrastructure Engineer, Cloud Data Engineer, DataOps Engineer, Data Pipeline Engineer, Devops Engineer or in a similar role. Proven experience with Azure cloud platforms and related architecture. Highly skilled in Python for data engineering, scripting, and automation. More ❯

We’re looking for an innovative Data Architect to lead the creation and rollout of a comprehensive Data Governance Strategy, alongside the implementation of a data platform. This role will be instrumental in ensuring organizational data is accurate, secure, compliant, and strategically utilized to unlock business value. This is a new role … created due to growth, offering an exciting opportunity to shape and lead our data governance and platform strategy from the ground up. Key responsibilities Design and implement a comprehensive data governance framework. Define and maintain data governance policies, standards, and procedures. Establish and lead a cross-functional Data Governance Council. Collaborate with … data owners, stewards, and custodians across departments to ensure alignment and accountability. Develop and enforce data quality standards, metrics, and monitoring processes. Ensure compliance with data protection regulations such as GDPR, CCPA, and other relevant standards. Evaluate, select, and implement data governance tools (e.g. data catalogues, data quality More ❯

Hatfield based (with min of 2 days in the office) Competitive basic salary + Core & flexible benefits Join Data Management within our Data & Analytics function. We operate across four pillars: Data Management, Data Strategy, Data Governance and Data Delivery. We work in agile, delivering in sprints with modern … software engineering and DevOps practices to ship reliable data products at pace. The team owns the Group Data Warehouse, a global data platform that supports Computacenters combined reporting requirements across the world. In Data Management, you will turn Data Demand into prioritised backlog requirements and partner with business units to … deliver maximum customer value from data. If you enjoy practical delivery, measurable outcomes and helping teams make better decisions with trusted data, you will fit right in. What youll do As a Senior Data Consultant, you will play a key role in shaping and delivering data solutions that support business strategy. You will lead More ❯