951 to 975 of 1,184 Data Pipeline Jobs in the UK excluding London

Senior Data Engineer - Commodities - 140,000k + bonuses You don’t need any prior buy-side experience to join this firm, where you’ll join this new team within this established firm whose mission is to bridge the gap across technology and trading. This global energy and commodities firm is seeking an experienced Senior Data Engineer … the business users to understand what they need, before building out the application. You’ll see the impact you’re having, daily. Key responsibilities include building and optimising data pipelines, managing cloud-based storage and processing systems, and partnering with analysts and data scientists to deliver production-ready data solutions. They are looking for … an experience data engineer with strong skills in Python, SQL, and modern cloud technologies to take a leading role in a forward-thinking, data-driven organisation. No up-to-date CV required. More ❯

to humanity's enduring challenges. We are looking for a Senior DevSecOps Engineer to join the Pathogen Programme at EIT. In this role, you'll help ensure our data platform is built to the highest standards, with a strong emphasis on automation across the development lifecycle. You'll work closely with engineers to deploy data pipelines … their workflows. You'll be responsible for maintaining infrastructure, designing secure automation pipelines, managing cloud environments, and ensuring security and compliance. You'll collaborate with cross-functional teams, data engineers, backend, and full-stack developers, to build robust, automated deployment pipelines across our environments. Key Responsibilities Design, implement, and maintain secure cloud infrastructure using Oracle Cloud Infrastructure (OCI … tools like Terraform to enable secure, repeatable deployments. Implement and manage CI/CD pipelines, focusing on automated security testing, deployment, and monitoring. Ensure all aspects of the data platform OCI infrastructure, data ingest pipelines, tool deployments, access controls, and monitoring are developed, tested, and deployed using automation best practices. Support bioinformaticians in building pipelines that More ❯

Must be a British Citizen** Join one of the UK’s leading technology and defence engineering consultancies — a team trusted by government and security clients to deliver advanced data, software, and AI-driven solutions that protect the country’s most critical systems. They’re expanding their Gloucester engineering function by 30+ people before Q1 next year, with opportunities … Python Engineers to work on projects that genuinely make a difference. What you’ll be doing Building secure, high-performance software underpinning security and defence operations Developing complex data pipelines, APIs, and AI tooling in modern Python environments Working in agile, multi-disciplinary teams alongside experts in software, data, and machine learning Delivering production-grade solutions … required Strong hands-on Python development (FastAPI, Flask, Django or similar) Solid understanding of CI/CD pipelines, automated testing, and modern DevOps practices Experience in secure systems, data engineering, or AI-focused environments UK Citizen – must be eligible and willing to undergo DV Clearance The Package Up to £70,000 + excellent benefits Hybrid model based from More ❯

to step into a broader architectural remit. You will play a key role in shaping cloud strategies, modernising legacy systems, and enabling advanced technologies such as generative AI, data analytics, and cloud-native applications. Key Responsibilities Design and implement scalable, secure, and high-performing AWS solutions. Lead cloud migration and modernisation initiatives, including mainframe transformation. Collaborate with cross … functional teams including data scientists, engineers, and developers. Provide technical leadership and guidance on AWS best practices and architecture patterns. Optimise cloud infrastructure for performance, cost, and security. Support the deployment of AI/ML models and data pipelines in production environments. Stay current with emerging AWS services and recommend improvements to cloud environments. Required Skills … and architecture. Ideal for cloud engineers or AWS consultants with deep service-level understanding. Desirable Experience Background in infrastructure or networking. Exposure to generative AI, AWS Contact Center, data and analytics, or mainframe modernisation. Multi-cloud or hybrid cloud experience (e.g. Azure, GCP). Understanding of interoperability across cloud platforms and on-prem environments. Additional Information Candidates must More ❯

major national programme. This is a fantastic opportunity to take ownership of building and leading a team delivering cutting-edge digital infrastructure that will transform how vital scientific data is accessed and shared across the UK and beyond. The role As Lead Developer, you’ll provide senior technical direction and hands-on expertise in designing and implementing scalable … digital infrastructure programme. Provide hands-on technical input, writing high-quality code and driving architecture decisions. Design and implement secure, scalable backend systems and cloud infrastructure. Build robust data pipelines to manage and mobilise large-scale datasets. Collaborate with external suppliers, partners, and stakeholders to align delivery. What we’re looking for: At least 5 years of experience More ❯

major national programme. This is a fantastic opportunity to take ownership of building and leading a team delivering cutting-edge digital infrastructure that will transform how vital scientific data is accessed and shared across the UK and beyond. The role As Lead Developer, you’ll provide senior technical direction and hands-on expertise in designing and implementing scalable … digital infrastructure programme. Provide hands-on technical input, writing high-quality code and driving architecture decisions. Design and implement secure, scalable backend systems and cloud infrastructure. Build robust data pipelines to manage and mobilise large-scale datasets. Collaborate with external suppliers, partners, and stakeholders to align delivery. What we’re looking for: At least 5 years of experience More ❯

major national programme. This is a fantastic opportunity to take ownership of building and leading a team delivering cutting-edge digital infrastructure that will transform how vital scientific data is accessed and shared across the UK and beyond. The role As Lead Developer, you’ll provide senior technical direction and hands-on expertise in designing and implementing scalable … digital infrastructure programme. Provide hands-on technical input, writing high-quality code and driving architecture decisions. Design and implement secure, scalable backend systems and cloud infrastructure. Build robust data pipelines to manage and mobilise large-scale datasets. Collaborate with external suppliers, partners, and stakeholders to align delivery. What we’re looking for: At least 5 years of experience More ❯

AWS, GCP, Azure) for ML workloads and APIs Setting up ML nodes for distributed training and local development Managing containerised environments (Docker, Kubernetes, Terraform) Optimising storage for big data pipelines supporting ML workloads Monitoring systems and responding to incidents, ensuring reliability and performance Working closely with ML engineers and researchers to integrate infra with production workloads What you … in automating deployments and orchestrating cloud environments Nice to have: Python (Jupyter, PyTorch), monitoring tools (Prometheus, Grafana), cloud databases (RDS, Aurora, Spanner), CI/CD tools (CircleCI), and data visualisation experience. This is a unique opportunity to join a visionary team redefining AI in 3D , with the chance to make a real impact at the cutting edge of More ❯

AWS, GCP, Azure) for ML workloads and APIs Setting up ML nodes for distributed training and local development Managing containerised environments (Docker, Kubernetes, Terraform) Optimising storage for big data pipelines supporting ML workloads Monitoring systems and responding to incidents, ensuring reliability and performance Working closely with ML engineers and researchers to integrate infra with production workloads What you … in automating deployments and orchestrating cloud environments Nice to have: Python (Jupyter, PyTorch), monitoring tools (Prometheus, Grafana), cloud databases (RDS, Aurora, Spanner), CI/CD tools (CircleCI), and data visualisation experience. This is a unique opportunity to join a visionary team redefining AI in 3D , with the chance to make a real impact at the cutting edge of More ❯

be doing: Lead the design and deployment of next-generation AI systems across Generative AI, Computer Vision, Forecasting, and Optimisation. Build and lead a world-class team of data scientists and ML engineers, setting technical direction and best practices across the company. Oversee the full AI lifecycle from experimentation and model development to robust ML deployment on Azure. … learning to enhance decision-making and automation. Translate complex research into tangible business outcomes, driving measurable value and competitive advantage. About you: Proven track record leading AI or data science teams in production supply chain or logistics environments. Deep expertise in Computer Vision and Generative AI, with strong understanding of modern ML architectures, model training, and deployment techniques. … 7+ years’ experience in applied ML, data science, or AI, including senior or leadership roles. Expert in Python, proficient with frameworks such as PyTorch, TensorFlow, FastAI, and experience with scikit-learn, MLflow, or ONNX. Solid engineering background—experienced in deploying and maintaining ML systems on Azure, including Azure Machine Learning, Azure Functions, Databricks, Docker, and Kubernetes. Experience with More ❯

be doing: Lead the design and deployment of next-generation AI systems across Generative AI, Computer Vision, Forecasting, and Optimisation. Build and lead a world-class team of data scientists and ML engineers, setting technical direction and best practices across the company. Oversee the full AI lifecycle from experimentation and model development to robust ML deployment on Azure. … learning to enhance decision-making and automation. Translate complex research into tangible business outcomes, driving measurable value and competitive advantage. About you: Proven track record leading AI or data science teams in production supply chain or logistics environments. Deep expertise in Computer Vision and Generative AI, with strong understanding of modern ML architectures, model training, and deployment techniques. … 7+ years’ experience in applied ML, data science, or AI, including senior or leadership roles. Expert in Python, proficient with frameworks such as PyTorch, TensorFlow, FastAI, and experience with scikit-learn, MLflow, or ONNX. Solid engineering background—experienced in deploying and maintaining ML systems on Azure, including Azure Machine Learning, Azure Functions, Databricks, Docker, and Kubernetes. Experience with More ❯

be doing: Lead the design and deployment of next-generation AI systems across Generative AI, Computer Vision, Forecasting, and Optimisation. Build and lead a world-class team of data scientists and ML engineers, setting technical direction and best practices across the company. Oversee the full AI lifecycle from experimentation and model development to robust ML deployment on Azure. … learning to enhance decision-making and automation. Translate complex research into tangible business outcomes, driving measurable value and competitive advantage. About you: Proven track record leading AI or data science teams in production supply chain or logistics environments. Deep expertise in Computer Vision and Generative AI, with strong understanding of modern ML architectures, model training, and deployment techniques. … 7+ years’ experience in applied ML, data science, or AI, including senior or leadership roles. Expert in Python, proficient with frameworks such as PyTorch, TensorFlow, FastAI, and experience with scikit-learn, MLflow, or ONNX. Solid engineering background—experienced in deploying and maintaining ML systems on Azure, including Azure Machine Learning, Azure Functions, Databricks, Docker, and Kubernetes. Experience with More ❯

heart of their organisation by identifying and delivering products and services that are right for their customers. Due to a recent merger, my client is looking for a Data Engineer to join their Data team to design, develop and test quality data engineering solutions and to challenge/improve processes, tools and approaches. There … is a potential for the role to be extended or to be moved into a permanent position in the future. Migration and integration of data into their new ecosystem Take responsibility for testing of solutions including the analysis of requirements, designs of test cases/scripts, preparing test date and creating/executing tests Strong experience as a … Data Engineer (migrating legacy systems onto AWS, building data pipelines) Strong Python experience AWS Glue, Redshift, Lambda, PySpark, Airflow SSIS or SAS experience (Desirable) Salary up to 57,500 + up to 20% bonus Hybrid working: once a fortnight in the office 28 days holiday plus bank holidays (option to buy and sell) Life assurance (6x More ❯

that this is a fully on site role, and candidates for this role are required to hold an active DV security clearance. Hexegic are looking for technically minded data and software engineers to work on an intriguing and exciting project, building and maintaining data pipelines and platforms for our customers in a military setting. Our growing … team ensure data driven projects can be rapidly prototyped and delivered in line with customer needs. About us Hexegic are a leading technical consultancy providing agile multi-disciplinary teams to high performing organisations. The company promises exciting, engaging and rewarding projects for those that are keen to develop and build a successful career. Core Responsibilities Liaise closely with … Strong python proficiency, or familiarity with similar languages API awareness for data retrieval and integration into processing pipelines Experience of stakeholder management and requirements capture Understanding of pipeline scheduling and health checks What’s in it for you? Base salary of £50,000-£70,000 £5000 a year professional development budget Wellness program 25 days annual leave More ❯

that this is a fully on site role, and candidates for this role are required to hold an active DV security clearance. Hexegic are looking for technically minded data and software engineers to work on an intriguing and exciting project, building and maintaining data pipelines and platforms for our customers in a military setting. Our growing … team ensure data driven projects can be rapidly prototyped and delivered in line with customer needs. About us Hexegic are a leading technical consultancy providing agile multi-disciplinary teams to high performing organisations. The company promises exciting, engaging and rewarding projects for those that are keen to develop and build a successful career. Core Responsibilities Liaise closely with … Strong python proficiency, or familiarity with similar languages API awareness for data retrieval and integration into processing pipelines Experience of stakeholder management and requirements capture Understanding of pipeline scheduling and health checks What’s in it for you? Base salary of £50,000-£70,000 £5000 a year professional development budget Wellness program 25 days annual leave More ❯

heart of their organisation by identifying and delivering products and services that are right for their customers. Due to a recent merger, my client is looking for a Data Engineer to join their Data team to design, develop and test quality data engineering solutions and to challenge/improve processes, tools and approaches. There … is a potential for the role to be extended or to be moved into a permanent position in the future. Main responsibilities Migration and integration of data into their new ecosystem Undertake review and assurance activity Provide team members with guidance on design, build and test activity Deliver solutions that meet business needs in a timely manner Take … for testing of solutions including the analysis of requirements, designs of test cases/scripts, preparing test date and creating/executing tests Requirements Strong experience as a Data Engineer (migrating legacy systems onto AWS, building data pipelines) Strong Python experience Tech stack experience required: AWS Glue, Redshift, Lambda, PySpark, Airflow SSIS or SAS experience (Desirable More ❯

heart of their organisation by identifying and delivering products and services that are right for their customers. Due to a recent merger, my client is looking for a Data Engineer to join their Data team to design, develop and test quality data engineering solutions and to challenge/improve processes, tools and approaches. There … is a potential for the role to be extended or to be moved into a permanent position in the future. Main responsibilities Migration and integration of data into their new ecosystem Undertake review and assurance activity Provide team members with guidance on design, build and test activity Deliver solutions that meet business needs in a timely manner Take … for testing of solutions including the analysis of requirements, designs of test cases/scripts, preparing test date and creating/executing tests Requirements Strong experience as a Data Engineer (migrating legacy systems onto AWS, building data pipelines) Strong Python experience Tech stack experience required: AWS Glue, Redshift, Lambda, PySpark, Airflow SSIS or SAS experience (Desirable More ❯

heart of their organisation by identifying and delivering products and services that are right for their customers. Due to a recent merger, my client is looking for a Data Engineer to join their Data team to design, develop and test quality data engineering solutions and to challenge/improve processes, tools and approaches. There … is a potential for the role to be extended or to be moved into a permanent position in the future. Main responsibilities Migration and integration of data into their new ecosystem Undertake review and assurance activity Provide team members with guidance on design, build and test activity Deliver solutions that meet business needs in a timely manner Take … for testing of solutions including the analysis of requirements, designs of test cases/scripts, preparing test date and creating/executing tests Requirements Strong experience as a Data Engineer (migrating legacy systems onto AWS, building data pipelines) Strong Python experience Tech stack experience required: AWS Glue, Redshift, Lambda, PySpark, Airflow SSIS or SAS experience (Desirable More ❯

Position: Data EngineerLocation: Central London or Bristol (3 days per week onsite)Salary: £600 to £650 p.d inside IR35Security Clearance: SC Clearance Required About the Role A leading IT consultancy delivering innovative solutions to the UK public sector is seeking an experienced Data Engineer to support a high-profile programme. You'll collaborate with cross-functional … teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation Contribute to … Docker/Kubernetes) - experience in this area is desirable, not essential Operate within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/ More ❯

and over £20M in annual sales, they're now investing heavily in their technology to unlock the next phase: scaling from 150 sites to 5,000+, powered by data, AI, and automation. The Role You'll be a foundational member of their engineering team. This is a high-impact individual contributor role, ideal for someone who enjoys building … the ground up, solving real-world problems, and owning architecture decisions in a fast-moving environment. You’ll work across the stack, with a focus on backend systems, data infrastructure, and internal tools. What You’ll Be Doing Implementing AI tooling across all their products focusing on data ingestion and aggregation Lead the design and implementation … of scalable, microservices-based systems Own and evolve their core platform architecture, including APIs, internal tools, and data pipelines Work with large, varied data sets from third-party sources (e.g. partner platforms, supply chains, operational data) Help turn data into insights and recommendations using AI/ML tooling Tackle technical debt and More ❯

and over £20M in annual sales, they're now investing heavily in their technology to unlock the next phase: scaling from 150 sites to 5,000+, powered by data, AI, and automation. The Role You'll be a foundational member of their engineering team. This is a high-impact individual contributor role, ideal for someone who enjoys building … the ground up, solving real-world problems, and owning architecture decisions in a fast-moving environment. You’ll work across the stack, with a focus on backend systems, data infrastructure, and internal tools. What You’ll Be Doing Implementing AI tooling across all their products focusing on data ingestion and aggregation Lead the design and implementation … of scalable, microservices-based systems Own and evolve their core platform architecture, including APIs, internal tools, and data pipelines Work with large, varied data sets from third-party sources (e.g. partner platforms, supply chains, operational data) Help turn data into insights and recommendations using AI/ML tooling Tackle technical debt and More ❯

and over £20M in annual sales, they're now investing heavily in their technology to unlock the next phase: scaling from 150 sites to 5,000+, powered by data, AI, and automation. The Role You'll be a foundational member of their engineering team. This is a high-impact individual contributor role, ideal for someone who enjoys building … the ground up, solving real-world problems, and owning architecture decisions in a fast-moving environment. You’ll work across the stack, with a focus on backend systems, data infrastructure, and internal tools. What You’ll Be Doing Implementing AI tooling across all their products focusing on data ingestion and aggregation Lead the design and implementation … of scalable, microservices-based systems Own and evolve their core platform architecture, including APIs, internal tools, and data pipelines Work with large, varied data sets from third-party sources (e.g. partner platforms, supply chains, operational data) Help turn data into insights and recommendations using AI/ML tooling Tackle technical debt and More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯