1 to 25 of 825 Permanent Data Pipeline Jobs in the UK

impact on those who deserve compensation, and this role suits a candidate who can spearhead solutions from the ground up to take from ideation to reality so that data is an enabler to everything IBCA does. Opportunities to move to the new Infected Blood Compensation Authority (IBCA). At the moment all IBCA roles are hosted by Cabinet … support and practical guidance to find a new role, whether in the Civil Service, another Arms Length Body (ALB), or an external employer. This role will lead our data platform architecture and modelling team within the Data Operations arm of the IBCA Data Directorate. The Data Operations team is responsible for developing … and running safe and secure data solutions that provide a single source of truth for those going through their compensation journey. They are designing and building a new data platform using Amazon Web Services (AWS) and data management and intelligence products using Databricks and Quantexa. We are taking a product-centric approach treating data More ❯

About Us: ATREIDES is a leader in advanced geospatial technologies, big data solutions, and data analytics. We specialize in providing cutting-edge software platforms that enable our clients to process, analyze, and visualize large-scale geospatial and sensor data for critical decision-making. Our mission is to deliver innovative, scalable, and high-performance platforms … that help organizations unlock the full potential of their data for various industries, including defense, national security, environmental monitoring, and urban planning. We are seeking a highly skilled Senior Data Engineer with a strong focus on big data, data analytics, and geospatial data intelligence to join our engineering team. In … and technologies (AWS, Azure, GCP) for big data processing and platform deployment. Strong knowledge of data warehousing, data lakes, and data pipeline design for large-scale data integration and storage. Familiarity with machine learning and AI techniques for data analytics (e.g., classification, regression, clustering, anomaly detection). More ❯

Description Data System Reliability Engineer (dSRE) Role Overview: A crucial role in CME's Cloud data transformation, the data SRE will be aligned to data product pods ensuring that our data infrastructure is reliable, scalable, and efficient as the GCP data footprint expands rapidly. Accountabilities: Automate data tasks on GCP Work with data domain owners, data scientists, and other stakeholders to ensure effective data consumption on GCP Design, build, secure, and maintain data infrastructure, including data pipelines, databases, data warehouses, and data processing platforms on GCP Measure and monitor the quality … of data on GCP data platforms Implement robust monitoring and alerting systems to proactively identify and resolve issues in data systems. Respond to incidents promptly to minimize downtime and data loss. Develop automation scripts and tools to streamline data operations and make them scalable to accommodate growing data More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … data engineering projects. You will work closely with cross-functional teams and contribute to the strategic direction of our data initiatives. RESPONSIBILITIES Data Pipeline Development: Lead the design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such More ❯

resources. We help companies make strategic, sustainable decisions by bringing transparency to markets across the world. We create a comprehensive view of commodities markets, providing companies with the data and intelligence to successfully navigate across global value chains every day. Our customers benefit from instant access to price assessments, reports and forecasts, a dedicated news channel and supply … within a dynamic market-leading global business. We have a supportive culture with a keen focus on innovation, technical excellence, career development and mutual support. Responsibilities Data Pipeline Development:Design, develop, and optimize robust data pipelines and ETL processes to ensure efficient data flow and integration Data Infrastructure Management:Manage and … possible data, to create high value analytical services Ownership of assigned data products, including data model design, end-to-end data pipeline delivery, data product quality monitoring, requirements analysis and issue resolution Enthusiastic attitude to explore and implement advanced data technologies and tools Working with other tech More ❯

Job summary Are you looking for a pivotal role, where you can leverage your extensive experience in data engineering and architecture to design and implement robust data solutions that drive business insights and decision-making? We are looking for a Principal Data Engineer to lead a growing team and to own and help deliver … our Digital and Data strategy. Your leadership will be instrumental in setting technical direction and mentoring engineers to optimize our data pipelines and infrastructure. You will collaboratively develop and maintain data models, architecture, and integration strategies so that the organisation can use its data well and safely. You are equally as comfortable …/ELT architecture at scale. Advanced programming skills in SQL and Python and a collaborative, CI/CD approach to pushing code; experience with orchestration and data pipeline tools. Experience with implementing data governance frameworks and ensuring compliance with regulatory standards. Familiarity with ML/LLM data needs and setup Effective communication and More ❯

Job summary We are looking for an experienced Senior Data Engineer with … strong technical expertise and innovative mindset to drive our team forward. Main duties of the job Data Engineer will be involved in the entire data pipeline from data acquisition to data extraction, ensuring that the data is accurate, reliable, and accessible, and fit-for-purpose for business intelligence, analytics … and data science functions. Data Engineer will be collaborating with cross-functional teams to support the delivery of impactful analytics work for our clients. About us We are the official research support team for the NHS Benchmarking Network, working with over 300 healthcare organisations across the UK to identify opportunities for service development and quality improvement. More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … modelling, ETL (Extract, Transform, Load) processes, and big data technologies it becomes possible to develop robust and reliable data solutions. RESPONSIBILITIES Data Pipeline Development: Design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such as databricks More ❯

Job summary Are you a data engineer with a passion for public service? Then maybe you're looking for us! We are seeking an experienced and skilled Advanced Data Engineer to be part of a growing team that delivers on our mission of protecting and improving the health and wellbeing of the people of Wales. You … ll be involved in improving our data practices, cataloguing our data, automating our data processing, analytical pipelines and developing our overall data maturity. Working with us you'll collaborate with other technical specialists (such as developers, database administrators, data analysts and data scientists) on the development, implementation, maintenance … ELT at scale. Advanced programming skills in SQL and Python [LN3][FB4]and a collaborative, CI/CD approach to pushing code; experience with orchestration and data pipeline tools. A user-focused approach to data delivery, ensuring that outputs are accessible, actionable and grounded in real-world context Effective communication and stakeholder engagement skills, with More ❯

consistency, and proactively collaborating with upstream and downstream teams to enable seamless data flow across the organization. In this role, you will not only troubleshoot and resolve pipeline issues but also contribute to enhancing data architecture, implementing best practices in data governance and security, and ensuring the scalability and performance of data … stakeholders This position requires client presence between 25%-50% of the time per month at the client's office, which is located in London. Key Responsibilities: Data Pipeline Development & Maintenance Build, maintain, and optimize scalable ETL/ELT pipelines using tools such as Dagster, or similar. Ensure high data availability, reliability, and consistency through rigorous … data validation and monitoring practices. Collaborate with cross-functional teams to align data pipeline requirements with business objectives and technical feasibility. Automate data workflows to improve operational efficiency and reduce manual intervention. Data Integrity & Monitoring Perform regular data consistency checks, identifying and resolvinganomalies or discrepancies. Implement robust monitoring frameworks More ❯

Report to: Director of Data Science Location : Hybrid (2 days/week in office, London Holborn) Salary : 60-65k DOE About Us The data team at Times Higher Education works with real-world data inputs from a number of external sources, and our work is reliant on this data being easily … also need to be shared across the company and to external parties, for a growing number of applications. Your role will be focused on supporting the delivery of data to our various internal stakeholders, within and outside the data team. You will work closely with our Dir. Of Data Science, Data Scientists … analyst/scientist with strong data engineering skills), who can work on ad-hoc data delivery as well as help strengthen our whole pipeline and governance, providing expert input and support to the team. Responsibilities Support Data Delivery team in data collection and provision processes Support Consultancy team in More ❯

legacy processes for performance, maintainability, and scalability Implement CI/CD best practices for data transformation using dbt Cloud and Git-based workflows Monitor, troubleshoot, and optimize pipeline performance and ensure data integrity across data platform Contribute to defining and improving data engineering standards, documentation, and best practices Provide technical mentorship … of legacy tools like Talend and Jenkins, and an understanding of their migration challenges Familiarity with version control (Git) and modern deployment workflows Proficiency in troubleshooting data pipeline failures and performance bottlenecks Experience with data pipeline orchestration (Airflow or similar is a plus) Nice-to-have experience: Exposure to other parts of the modern … data stack (e.g., Fivetran, dbt Metrics Layer, Tableau) Experience in CI/CD pipeline tooling and Infrastructure as Code (e.g., GitHub Actions, Terraform) Understanding of data governance, privacy, and compliance best practices (e.g., GDPR, SOC 2) About UnderwriteMe UnderwriteMe is an Insurtech software business and subsidiary of Pacific Life Re (PL Re), a global life More ❯

Senior Data Engineer - Job Description Senior Data Engineer - New Data Products page is loaded Senior Data Engineer - Job Description Senior Data Engineer - New Data Products Apply locations London, UK time type Full time posted on Posted 2 Days Ago job requisition id R14815 Job Title Senior Data … Familiarity with version control (Git) and modern deployment workflows Excellent problem-solving skills and ability to resolve data issues quickly and effectively Experience with data pipeline orchestration (Airflow or similar is a plus) Nice-to-have experience: Exposure to other parts of the modern data stack (e.g., Fivetran, dbt Metrics Layer, Tableau) Experience … in CI/CD pipeline tooling and Infrastructure as Code (e.g., GitHub Actions, Terraform) Familiarity with data governance, cataloging, and lineage tools Experience with the DataVault 2.0 methodology and implementing hub, satellite, and link structures Knowledge of Python or other scripting languages used in data engineering workflows Prior experience in an Agile/Scrum team More ❯

officially designated Posts. Our UK-based staff work in King Charles Street, London, Abercrombie House in East Kilbride and in Milton Keynes. These roles contribute to FCDOs Data Strategy to improve insights to the business by provisioning it with reliable and trustworthy data sources and improving theirdeliveryin a secure manner in compliance with Data regulations. The team is primarily Azure focused but other Cloud technologies such as Oracle andAmazon Web Services (AWS) are usedin FCDO. The successful candidates will build complex data pipelines (both Extract Transform and Load ETL and Extract Load and Transform ELT ) in the Azure cloud platforms. You will work with structured and unstructured data, data lakes, data warehouses to service operational and analytical business needs. The successful candidates will: Work collaboratively with Enterprise Data Colleagues to develop pipelines and data product that aligns with data architecture and engineering best practice. Develop data assets and products that assist Power Business Intelligence (BI) colleagues to More ❯

officially designated Posts. Our UK-based staff work in King Charles Street, London, Abercrombie House in East Kilbride and in Milton Keynes. These roles contribute to FCDOs Data Strategy to improve insights to the business by provisioning it with reliable and trustworthy data sources and improving theirdeliveryin a secure manner in compliance with Data regulations. The team is primarily Azure focused but other Cloud technologies such as Oracle andAmazon Web Services (AWS) are usedin FCDO. The successful candidates will build complex data pipelines (both Extract Transform and Load ETL and Extract Load and Transform ELT ) in the Azure cloud platforms. You will work with structured and unstructured data, data lakes, data warehouses to service operational and analytical business needs. The successful candidates will: Work collaboratively with Enterprise Data Colleagues to develop pipelines and data product that aligns with data architecture and engineering best practice. Develop data assets and products that assist Power Business Intelligence (BI) colleagues to More ❯

time left to apply End Date: July 24, 2025 (13 days left to apply) job requisition id JR102754 We're undertaking a fast paced data transformation into Databricks at E.ON Next using best practice data governance and architectural principles, and we are growing our data engineering capability within the Data Team. As … part of our journey we're looking for a data architect to help bring our vision to life to design and review data models, iterate on data architecture, and ensure data is efficiently stored and managed within our enterprise data lakehouse. At E.ON Next we're looking for the most … useful formats. Able to explain data storage concepts and logical data structures i.e. data warehousing. Able to make recommendations for data pipeline design based on business requirements. Be open to new ways of working, and open to working with new technologies. Be able to set your own goals and take the More ❯

time left to apply End Date: July 24, 2025 (13 days left to apply) job requisition id JR102754 We're undertaking a fast paced data transformation into Databricks at E.ON Next using best practice data governance and architectural principles, and we are growing our data engineering capability within the Data Team. As … part of our journey we're looking for a data architect to help bring our vision to life to design and review data models, iterate on data architecture, and ensure data is efficiently stored and managed within our enterprise data lakehouse. At E.ON Next we're looking for the most … useful formats. Able to explain data storage concepts and logical data structures i.e. data warehousing. Able to make recommendations for data pipeline design based on business requirements. Be open to new ways of working, and open to working with new technologies. Be able to set your own goals and take the More ❯

time left to apply End Date: July 24, 2025 (13 days left to apply) job requisition id JR102754 We're undertaking a fast paced data transformation into Databricks at E.ON Next using best practice data governance and architectural principles, and we are growing our data engineering capability within the Data Team. As … part of our journey we're looking for a data architect to help bring our vision to life to design and review data models, iterate on data architecture, and ensure data is efficiently stored and managed within our enterprise data lakehouse. At E.ON Next we're looking for the most … useful formats. Able to explain data storage concepts and logical data structures i.e. data warehousing. Able to make recommendations for data pipeline design based on business requirements. Be open to new ways of working, and open to working with new technologies. Be able to set your own goals and take the More ❯

who will work with Head of Data and Enterprise architecture to build data engineering process in AWS using modern tech stack. Responsibilities Data Pipeline Development : Design, build, and maintain scalable data pipelines to ingest, process, and store large sets of financial data from various internal and external sources. Cloud … security, and performance of data pipelines and integrations. Documentation & Best Practices : Maintain clear, well-organised documentation for all data engineering projects, including data pipeline processes, cloud infrastructure, and codebase. Data Governance & Security : Implement and enforce data governance policies and security practices, ensuring compliance with financial regulations and protecting sensitive … financial data. Troubleshooting & Optimisation : Monitor and troubleshoot data pipeline performance, addressing bottlenecks or issues promptly to ensure smooth data flow and operation. Requirements Several years of experience in data engineering, preferably in the financial services or similar regulated industries. Strong understanding of data engineering concepts, including data modelling, ETL More ❯

The AI Engineer is a key role in the Data Platform Portfolio team, building the data platform, driving value from data across the business. With strong technical skills and business acumen to help turn millions of potential data points into actionable insights that can drive product improvements, make our customer acquisition more … efficient, improve our customer retention rates, and drive operating efficiencies across the business. The primary goals of the team are: To build and run a data platform which can create and deliver analytics to colleagues and deliver regulatory reporting Ingest and transform data from multiple systems, modelling data and engineering data marts … outliers. Proficient in designing and implementing ELT pipelines using tools like dbt, with strong knowledge of data warehousing, data lake concepts, and data pipeline optimization. Skilled in SQL for data manipulation, analysis, query optimisation, and database design. Artificial Intelligence and Machine Learning Understanding of machine learning algorithms (classification, regression, clustering) and More ❯

allowance based on a skills assessment Published on 10 July 2025 Deadline 30 July 2025 Location Croydon, Manchester, Sheffield About the job Job summary As a Lead Data Architect, you will create and uphold data architecture across critical Home Office systems, enabling secure, data-driven services that keep the country safe. Working under the … vision set by the Principal Data Architect, you'll design robust data models, metadata systems, and governance frameworks to serve diverse stakeholders. Your leadership will span strategic planning, mentoring junior data architects, and aligning technical decisions with business objectives. If you thrive on architecting solutions that integrate emerging technologies such as cloud-native platforms … data lakehouses, streaming data pipelines and AI-powered analytics while collaborating with multi disciplinary teams, this role offers you the chance to elevate how the Home Office exploits data for maximum public impact. Due to business requirements this post is available on a full time/flexible working basis. Where business needs allow, some More ❯

allowance based on a skills assessment Published on 10 July 2025 Deadline 30 July 2025 Location Croydon, Manchester, Sheffield About the job Job summary As a Lead Data Architect, you will create and uphold data architecture across critical Home Office systems, enabling secure, data-driven services that keep the country safe. Working under the … vision set by the Principal Data Architect, you'll design robust data models, metadata systems, and governance frameworks to serve diverse stakeholders. Your leadership will span strategic planning, mentoring junior data architects, and aligning technical decisions with business objectives. If you thrive on architecting solutions that integrate emerging technologies such as cloud-native platforms … data lakehouses, streaming data pipelines and AI-powered analytics while collaborating with multi disciplinary teams, this role offers you the chance to elevate how the Home Office exploits data for maximum public impact. Due to business requirements this post is available on a full time/flexible working basis. Where business needs allow, some More ❯

allowance based on a skills assessment Published on 10 July 2025 Deadline 30 July 2025 Location Croydon, Manchester, Sheffield About the job Job summary As a Lead Data Architect, you will create and uphold data architecture across critical Home Office systems, enabling secure, data-driven services that keep the country safe. Working under the … vision set by the Principal Data Architect, you'll design robust data models, metadata systems, and governance frameworks to serve diverse stakeholders. Your leadership will span strategic planning, mentoring junior data architects, and aligning technical decisions with business objectives. If you thrive on architecting solutions that integrate emerging technologies such as cloud-native platforms … data lakehouses, streaming data pipelines and AI-powered analytics while collaborating with multi disciplinary teams, this role offers you the chance to elevate how the Home Office exploits data for maximum public impact. Due to business requirements this post is available on a full time/flexible working basis. Where business needs allow, some More ❯

Senior Data Ops Data EngineerLeeds 2/3 days per week£55,000-£67,000+ Excellent Benefits18 Months FTC with long term potential Your New Company Our large public sector organisation are seeking a senior Data Platform DataOps Engineer to serve as our clients first DataOps specialist in a growing team of Data Engineers and DevOps professionals.In this pivotal role, you will focus on operationalising and automating their data lifecycle to ensure that data workflows perform with reliability and efficiency. You will integrate CI/CD data pipelines, streamline deployment processes, enforce robust data governance, and optimise operational costs within our Microsoft Azure … APIM. Proficiency in developing CI/CD data pipelines and strong programming skills in Python, SQL, Bash, and PySpark for automation. Strong aptitude for data pipeline monitoring and an understanding of data security practices such as RBAC and encryption. Implemented data and pipeline observability dashboards, ensuring high data More ❯

Data Engineer/Scientist Department: Consultancy Employment Type: Full Time Location: United Kingdom/Hybrid Description As a Data Engineer/Scientist at Actica, you will have the opportunity to design, build, and maintain data pipelines while developing advanced analytics solutions to unlock business problems for high-profile UK public sector organisations. Your expertise … will enable organisations to maximise the value of their data assets through robust data infrastructure and sophisticated analysis, playing a key role in nationally critical projects that make a real difference to people's everyday lives. Locations: London, Guildford, Bristol, M4 corridor Hybrid working Roles and Responsibilities Actica recognises that data engineering, analytics, and … data science are distinct but interconnected disciplines. Our data engineers focus on building and maintaining the data infrastructure that enables analytics and data science work, while our data scientists and analysts focus on deriving insights and developing models. At Actica, we're at the forefront of the UK government's More ❯