326 to 350 of 399 Data Pipeline Jobs in England

Lead Data Engineer - Python, SQL, GCP, Kubernetes, DBT, Airflow and BigQurey OB have partnered with a leading scale-up business in the AdTech space, where Data is at the forefront of everything they do, and they are currently hiring for a Lead Data Engineer to join their team and work on the development of … their Data platform. This will be a hands-on role but you will line manage a small team of 3 Engineers, where the long-term plan of this team is to grow and this is a great opportunity to progress in a scaling business. This role requires candidates to come with hands on experience working in Data heavy environments, with Real-Time Data Pipelines, Distributed Streaming Pipelines and strong knowledge of Cloud Environments. Lead Data Engineer - Python, SQL, GCP, Kubernetes, DBT, Airflow and BigQurey Key Skills and Experience: Python, SQL, GCP, Kubernetes, DBT, Airflow and BigQurey Prior experience working as a Lead Engineer or Tech Lead Pays £100k-£130k + bonus More ❯

This is a highly autonomous role, perfect for someone who thrives on building, managing, and continuously improving the systems that keep a fast-moving business running smoothly. From data pipelines to automation workflows, you will ensure every technical aspect of the business operates efficiently and strategically. Responsibilities: Own all technology systems and processes across the business. Design, build … and maintain data pipelines integrating POS, kitchen operations, labor, and inventory systems. Develop and manage dashboards and reporting tools for leadership and operations. Build and maintain low/no-code automation workflows to streamline operations. Manage API integrations between platforms. Identify, recommend, and implement new tech solutions that drive efficiency and growth. Ensure data accuracy, availability … security, and compliance. Partner with leadership to shape the company's tech and data strategy. Experience: Proven experience in data engineering, analytics, and reporting (SQL, Google BigQuery, Power BI, Looker, Tableau). Strong capability with API integrations and low/no-code tools (n8n, Zapier, Airtable, Make). Broad technical problem-solving skills, able to implement More ❯

reporting, P&L attribution, and compliance systems for Marketing IM Develop tools for real-time dashboards, scenario analysis, and risk monitoring (market, credit, operational) Build and maintain APIs, data pipelines, and automation for risk analytics and reporting Implement scalable solutions to monitor risks across inventories, derivatives, FX, and counterparty credit Enhance and integrate commercial off-the-shelf (COTS … Knowledge of market, credit, and operational risk, compliance, and trade lifecycle Proven in delivering complex risk and compliance solutions Strong leadership and stakeholder management skills Bonus: Python, Azure Data Factory/Databricks, data analysis, and COTS risk platforms (e.g., CubeLogic, deltaconX) Package 85,000 Base Salary 15% non-contributory pension Hybrid working (3 days per week More ❯

reporting, P&L attribution, and compliance systems for Marketing IM Develop tools for real-time dashboards, scenario analysis, and risk monitoring (market, credit, operational) Build and maintain APIs, data pipelines, and automation for risk analytics and reporting Implement scalable solutions to monitor risks across inventories, derivatives, FX, and counterparty credit Enhance and integrate commercial off-the-shelf (COTS … Knowledge of market, credit, and operational risk, compliance, and trade lifecycle Proven in delivering complex risk and compliance solutions Strong leadership and stakeholder management skills Bonus: Python, Azure Data Factory/Databricks, data analysis, and COTS risk platforms (e.g., CubeLogic, deltaconX) Package £85,000 Base Salary 15% non-contributory pension Hybrid working (3 days per week More ❯

demonstrating the power of our platform and ensuring our clients' success. Key Responsibilities Client Engagement & Discovery: Collaborate closely with enterprise clients to understand their business processes, pain points, data ecosystems, and strategic goals. Solution Design & Architecture: Design end-to-end solutions by mapping client requirements to our platform's offerings (e.g., Knowledge Hubs, Conversational AI, Real-time Insights … and Agentic Automation). Create detailed architecture documents, data flow diagrams, and technical specifications. MCP Workflow & Ontology Design: Architect the core intelligence for client solutions. This includes designing client-specific ontologies and defining the structure, logic, and composition of Model Context Protocols (MCPs) to automate complex reasoning and workflows. Technical Leadership: Serve as the deep technical expert on … the platform for both clients and internal teams. Provide guidance on best practices for data integration, RAG implementation, and agentic automation. Proof of Concept & Prototyping: Lead the technical design and development of Proof of Concept (PoC) projects to demonstrate the feasibility and value of our solutions for specific client use cases. Cross-functional Collaboration: Work closely with our More ❯

solutions, collaborating with multidisciplinary teams to deliver secure, scalable, and innovative technology. Key Responsibilities Design, develop, and deploy AI/ML models for real-world applications. Collaborate with data scientists, engineers, and stakeholders to deliver robust solutions. Ensure all work complies with security and governance standards. Contribute to the development of AI best practices and frameworks. Essential Skills … Active SC Clearance (must be current) Strong experience in AI/ML engineering (Python, TensorFlow, PyTorch, etc.) Experience with cloud platforms (AWS, Azure, or GCP) Solid understanding of data pipelines, APIs, and MLOps Excellent problem-solving and communication skills More ❯

most dynamic and complex markets in the world. This is a chance to build high-performance, low-latency, and cloud-native systems while tackling challenges in fraud detection, data engineering, and market integrity . What you'll do Design, implement, and optimise algorithms for detecting manipulation, fraud, and compliance breaches in real-time and batch data. Build and … scale high-volume data pipelines handling billions of daily events across distributed systems. Work with cutting-edge tech including Java, Kafka, Spark, Kubernetes, Clickhouse, Snowflake, Redis . Collaborate with quants, data scientists, and compliance experts to improve surveillance strategies. Contribute to system architecture for low-latency monitoring and high-throughput processing . Continuously improve data quality, reliability, and query optimisation at scale. What we're looking for 5+ years' experience in software engineering , ideally in financial markets, market surveillance, or large-scale data systems . Strong skills in Java (or similar: Kotlin, C#, C++) ; Python is a plus. Proven experience with real-time, low-latency systems and distributed data pipelines More ❯

pro rata for 6 months) Contract: Fixed Term, 6 months Location: Hybrid - Hull (3 days onsite per week) Are you a SQL Developer who thrives on solving complex data challenges and making systems smarter, faster, and more efficient? This 6-month FTC is a chance to get hands-on with cutting-edge Industry 4.0 and Smart Manufacturing projects … where your work will directly impact real-time operations. What youll be doing: Developing and optimising SQL Server solutions to support real-time data feeds from machines, sensors, and production lines Managing and enhancing time series data pipelines to power predictive analytics and performance monitoring Working on push notifications, protocols, and query optimisation to keep data fast, clean, and actionable Driving the migration to Microsoft Fabric, with a focus on KQL (Azure Data Explorer/Fabric Real-Time Analytics) Collaborating with cross-functional teams in an Agile environment to deliver scalable, high-value solutions Supporting reporting and analytics - Power BI now, Microsoft Fabric in the future What were looking for: 5+ years More ❯

pro rata for 6 months) Contract: Fixed Term, 6 months Location: Hybrid - Hull (3 days onsite per week)Are you a SQL Developer who thrives on solving complex data challenges and making systems smarter, faster, and more efficient? This 6-month FTC is a chance to get hands-on with cutting-edge Industry 4.0 and Smart Manufacturing projects … where your work will directly impact real-time operations. What you’ll be doing: Developing and optimising SQL Server solutions to support real-time data feeds from machines, sensors, and production lines Managing and enhancing time series data pipelines to power predictive analytics and performance monitoring Working on push notifications, protocols, and query optimisation to keep … data fast, clean, and actionable Driving the migration to Microsoft Fabric, with a focus on KQL (Azure Data Explorer/Fabric Real-Time Analytics) Collaborating with cross-functional teams in an Agile environment to deliver scalable, high-value solutions Supporting reporting and analytics - Power BI now, Microsoft Fabric in the future What we’re looking for More ❯

pro rata for 6 months) Contract: Fixed Term, 6 months Location: Hybrid - Hull (3 days onsite per week) Are you a SQL Developer who thrives on solving complex data challenges and making systems smarter, faster, and more efficient? This 6-month FTC is a chance to get hands-on with cutting-edge Industry 4.0 and Smart Manufacturing projects … where your work will directly impact real-time operations. What you’ll be doing: Developing and optimising SQL Server solutions to support real-time data feeds from machines, sensors, and production lines Managing and enhancing time series data pipelines to power predictive analytics and performance monitoring Working on push notifications, protocols, and query optimisation to keep … data fast, clean, and actionable Driving the migration to Microsoft Fabric, with a focus on KQL (Azure Data Explorer/Fabric Real-Time Analytics) Collaborating with cross-functional teams in an Agile environment to deliver scalable, high-value solutions Supporting reporting and analytics - Power BI now, Microsoft Fabric in the future What we’re looking for More ❯

scalable APIs (Flask/FastAPI) for internal and external tools. Partner with business users to gather requirements and iterate quickly on solutions. Implement testing frameworks, version control, and data validation processes. Integrate with market data, storage systems, and reporting platforms. Required Skills 5+ years' Python development experience, ideally within trading, risk, or financial services. Strong background … with Pandas, NumPy, and data science libraries . Hands-on experience with Flask or FastAPI for API development. Proficiency in time-series data manipulation and back-testing methods. Experience working with large datasets and real-time data pipelines . Strong performance optimisation and debugging skills. Confident communicator with the ability to liaise directly with … business stakeholders. Desirable Skills Experience in commodities indices, structured products, or index engineering . Knowledge of market data feeds (Bloomberg, Refinitiv, etc). Familiarity with containerisation (Docker) and cloud platforms (AWS/GCP). Database experience (SQL/NoSQL) and use of modern data formats (Parquet, HDF5). Contract: Initial term with possible extension Day Rate More ❯

About the Role Outside IR35 £200 - £300Initial 6 month contract - high possibility of 18 monthsOur client is hiring a Data Scientist – AI/ML Engineer to join our Core AI Services team. You’ll be at the forefront of designing and scaling cloud-native AI platform services and APIs that enable enterprise-wide AI adoption. Working within a … native services on Microsoft Azure , including Azure Functions, AKS, and API Management. Develop secure, well-documented RESTful APIs and SDKs , managing their full lifecycle. Deliver end-to-end data pipelines , orchestration, and service APIs. Deploy and manage AI models and tooling for enterprise-scale adoption. Embed security best practices around authentication, authorization, and data privacy. Participate … learning and innovation. Skills & Experience 10+ years in software engineering, with strong experience building cloud-native or platform services using Microsoft and .NET . 4+ years in a Data Scientist or AI/ML Engineer role. Proficiency in Python , including libraries like pandas , scikit-learn , TensorFlow , and PyTorch . Deep understanding of ML algorithms : regression, classification, clustering, and More ❯

Introduce and manage observability, monitoring, and evaluation frameworks for ML and AI services. Enable auto-evaluation of prompts and models against domain-specific requirements. Build Python-based microservices, data pipelines, and serverless functions. Collaborate with stakeholders to translate data and AI requirements into scalable solutions. Essential Experience & Skills 5+ years' engineering experience, with at least … years in ML Ops, Data Engineering, or AI infrastructure. Strong Python engineering skills (Pandas, Numpy, Jupyter, FastAPI, SQLAlchemy). Expertise in AWS services (certification desirable). Proven experience deploying and supporting LLMs in production. Strong understanding of LLM fine-tuning (PyTorch, TensorFlow, Hugging Face Trainer, etc.). Experience with ML tooling (e.g. SageMaker, LangChain/LangSmith, MLflow, Dataiku … in Generative AI trends and best practices. Desired Skills Experience with chatbots and conversational AI (voice or text). Familiarity with Terraform, Helm, Kubernetes, or Postgres. Exposure to Data Science, NLP, Explainable AI (XAI). Real-world delivery of Generative AI solutions, especially LLM-driven applications. *Rates depend on experience and client requirements More ❯

to humanity's enduring challenges. We are looking for a Senior DevSecOps Engineer to join the Pathogen Programme at EIT. In this role, you'll help ensure our data platform is built to the highest standards, with a strong emphasis on automation across the development lifecycle. You'll work closely with engineers to deploy data pipelines … their workflows. You'll be responsible for maintaining infrastructure, designing secure automation pipelines, managing cloud environments, and ensuring security and compliance. You'll collaborate with cross-functional teams, data engineers, backend, and full-stack developers, to build robust, automated deployment pipelines across our environments. Key Responsibilities Design, implement, and maintain secure cloud infrastructure using Oracle Cloud Infrastructure (OCI … tools like Terraform to enable secure, repeatable deployments. Implement and manage CI/CD pipelines, focusing on automated security testing, deployment, and monitoring. Ensure all aspects of the data platform OCI infrastructure, data ingest pipelines, tool deployments, access controls, and monitoring are developed, tested, and deployed using automation best practices. Support bioinformaticians in building pipelines that More ❯

to step into a broader architectural remit. You will play a key role in shaping cloud strategies, modernising legacy systems, and enabling advanced technologies such as generative AI, data analytics, and cloud-native applications. Key Responsibilities Design and implement scalable, secure, and high-performing AWS solutions. Lead cloud migration and modernisation initiatives, including mainframe transformation. Collaborate with cross … functional teams including data scientists, engineers, and developers. Provide technical leadership and guidance on AWS best practices and architecture patterns. Optimise cloud infrastructure for performance, cost, and security. Support the deployment of AI/ML models and data pipelines in production environments. Stay current with emerging AWS services and recommend improvements to cloud environments. Required Skills … and architecture. Ideal for cloud engineers or AWS consultants with deep service-level understanding. Desirable Experience Background in infrastructure or networking. Exposure to generative AI, AWS Contact Center, data and analytics, or mainframe modernisation. Multi-cloud or hybrid cloud experience (e.g. Azure, GCP). Understanding of interoperability across cloud platforms and on-prem environments. Additional Information Candidates must More ❯

business units. ️ Design AI-based architectures and recommend appropriate tools, technologies, and methodologies ️ Collaborate with business leaders to understand objectives and translate them into scalable AI models ️ Architect data pipelines to support machine learning workflows and model deployments ️ Evaluate and implement deep learning, computer vision, NLP, and predictive modeling solutions ️ Conduct risk assessments on AI implementations and mitigate … compliance or ethical concerns ️ Partner with DevOps to integrate AI models into production systems with CI/CD ️ Define enterprise AI standards and ensure reusability of components ️ Mentor data scientists and engineers in applying best practices ️ Present technical documentation and demos to internal and external stakeholders ️ Build prototypes to validate AI use cases and accelerate adoption ️ Ensure AI More ❯

major national programme. This is a fantastic opportunity to take ownership of building and leading a team delivering cutting-edge digital infrastructure that will transform how vital scientific data is accessed and shared across the UK and beyond. The role As Lead Developer, you'll provide senior technical direction and hands-on expertise in designing and implementing scalable … digital infrastructure programme. Provide hands-on technical input, writing high-quality code and driving architecture decisions. Design and implement secure, scalable backend systems and cloud infrastructure. Build robust data pipelines to manage and mobilise large-scale datasets. Collaborate with external suppliers, partners, and stakeholders to align delivery. What we're looking for: At least 5 years of experience More ❯

major national programme. This is a fantastic opportunity to take ownership of building and leading a team delivering cutting-edge digital infrastructure that will transform how vital scientific data is accessed and shared across the UK and beyond. The role As Lead Developer, you'll provide senior technical direction and hands-on expertise in designing and implementing scalable … digital infrastructure programme. Provide hands-on technical input, writing high-quality code and driving architecture decisions. Design and implement secure, scalable backend systems and cloud infrastructure. Build robust data pipelines to manage and mobilise large-scale datasets. Collaborate with external suppliers, partners, and stakeholders to align delivery. What we're looking for: At least 5 years of experience More ❯

Radbroke (Hybrid - 2 days/week in office) 6 Months + Umbrella Only - Inside IR35 About the Role We are seeking a highly skilled and experienced Senior AWS Data & ML Engineer to join our team in Radbroke. This hybrid role offers the opportunity to work on cutting-edge machine learning and data engineering projects, leveraging the … latest cloud technologies and MLOps practices. Required/Primary Skills: AWS Data Engineering ML Engineering ML-Ops ECS, Sagemaker Gitlab Jenkins CI/CD AI Lifecycle Experience in front-end development (HTML, Stream-lit, Flask Familiarity with model deployment and monitoring in cloud environments (AWS). Understanding of machine learning lifecycle and data pipelines. Proficiency with … Python, Pyspark, Big-data ecosystems Hands-on experience with MLOps tools (e.g., MLflow, Airflow, Docker, Kubernetes) Secondary Skills Experience with RESTful APIs and integrating backend services All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. More ❯

Radbroke (Hybrid - 2 days/week in office) 6 Months + Umbrella Only - Inside IR35 About the Role We are seeking a highly skilled and experienced Senior AWS Data & ML Engineer to join our team in Radbroke. This hybrid role offers the opportunity to work on cutting-edge machine learning and data engineering projects, leveraging the … latest cloud technologies and MLOps practices. Required/Primary Skills: AWS Data Engineering ML Engineering ML-Ops ECS, Sagemaker Gitlab Jenkins CI/CD AI Lifecycle Experience in front-end development (HTML, Stream-lit, Flask Familiarity with model deployment and monitoring in cloud environments (AWS). Understanding of machine learning lifecycle and data pipelines. Proficiency with … Python, Pyspark, Big-data ecosystems Hands-on experience with MLOps tools (e.g., MLflow, Airflow, Docker, Kubernetes) Secondary Skills Experience with RESTful APIs and integrating backend services All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. More ❯

About the Role: The Capital IQ Solutions Data Science team supports the S&P Capital IQ Pro platform with innovative Data Science and Machine Learning solutions, utilizing the most advanced NLP Generative AI models. This role presents a unique opportunity for hands-on ML/NLP/Gen AI/LLM scientists and engineers to advance … synthetic evaluation methods and metrics. Deploy NLP models while ensuring low latency, reliability, and scalability. Explore new methods for prompt engineering, model fine-tuning, optimization, document embeddings, and data chunking. Collaborate closely with product teams, business stakeholders, and engineers to ensure seamless integration of NLP models into production systems. Troubleshoot complex issues related to machine learning model development … and data pipelines, developing innovative solutions as needed. Actively research and identify the latest relevant methods and technologies in the field. What We're Looking For: Basic Required Qualifications: A degree in Computer Science, Mathematics, Statistics, Engineering or a related field. A solid understanding of Machine Learning and Deep Learning methods, along with their mathematical foundations. At least More ❯

rebalancing logic Attribution and decomposition analytics Build robust APIs using FastAPI or Flask Collaborate directly with business users to gather requirements and iterate quickly Implement testing, versioning, and data validation processes Ensure integration with internal data sources and reporting tools Required Skills 5+ years of Python development experience, ideally in financial services or commodities Strong proficiency … in Pandas , NumPy , and time-series data manipulation Proven experience with FastAPI or Flask Familiarity with large datasets and real-time data pipelines Excellent debugging, profiling, and performance optimization skills Strong communication skills and ability to work with non-technical stakeholders Preferred Skills Experience with commodities indices or structured products Knowledge of market data More ❯

the unrestricted right to work in the UK - this organisation is not able to offer sponsorship. Are you a skilled SQL Server Developer looking to take ownership of data solutions in a dynamic environment? This is a fantastic opportunity to design and implement advanced SQL queries, build robust data pipelines, and deliver impactful reporting using Power … BI. Key Responsibilities: Develop and optimise complex SQL queries and stored procedures. Design and implement scalable data pipelines and ETL processes using SSIS. Create and maintain insightful reports and dashboards using Power BI or SSRS. Collaborate with cross-functional teams to understand data requirements and deliver solutions that drive business decisions. Required Skills: Strong SQL Server … development experience. Proficiency with SSIS for ETL and data integration. Experience with reporting tools such as SSRS or Power BI. What's on Offer: Competitive salary up to £50,000. Hybrid working model - 2-3 days per week on-site in Cambridge. Opportunity to work on meaningful data projects with real business impact. To apply for More ❯

Manager, you will be responsible for the pricing of multiple motor and home products, including the following core activities: Maintaining and improving behavioural models, prioritising a range of data science techniques. Advance the adoption of data science & statistical techniques. Develop reporting structures to monitor pricing performance in an automated fashion. Develop innovative solutions to improve rating … increasingly competitive insurance market. Key Responsibilities: Maintain and improve the pricing process to meet business requirements, including; Development of market leading behavioural modelling using a combination of traditional & data science techniques Improve sophistication within the optimisation of Markerstudy’s direct brands Support delivery of pricing & underwriting business plan across Motor & Home portfolios Support development of new product offerings … Advance the adoption of the latest data science & statistical techniques across pricing & underwriting Research and leverage new and existing data sources; capturing and explaining trends with innovative data features Clearly communicate results and proposed actions to key decision makers across the business Champion and facilitate automation of reporting, data pipelines and quality More ❯

Manager, you will be responsible for the pricing of multiple motor and home products, including the following core activities: Maintaining and improving behavioural models, prioritising a range of data science techniques. Advance the adoption of data science & statistical techniques. Develop reporting structures to monitor pricing performance in an automated fashion. Develop innovative solutions to improve rating … increasingly competitive insurance market. Key Responsibilities: Maintain and improve the pricing process to meet business requirements, including; Development of market leading behavioural modelling using a combination of traditional & data science techniques Improve sophistication within the optimisation of Markerstudy's direct brands Support delivery of pricing & underwriting business plan across Motor & Home portfolios Support development of new product offerings … Advance the adoption of the latest data science & statistical techniques across pricing & underwriting Research and leverage new and existing data sources; capturing and explaining trends with innovative data features Clearly communicate results and proposed actions to key decision makers across the business Champion and facilitate automation of reporting, data pipelines and quality More ❯