251 to 275 of 338 Data Pipeline Jobs in the UK excluding London

building the AI platform of choice for modern dealmakers. Were a fast-growing start-up where ideas move quickly from concept to product. Our technology spans backend, frontend, data, and AI, giving our team real scope to shape the future of the product and the company itself. If youre excited by solving complex problems end-to-end, working … the companys journey towards achieving its business milestones. Responsibilities (including but not limited to): Backend APIs (Python/FastAPI): Build reliable, secure services that power AI features and data retrieval at scale. RAG & vector search: Design, implement and iterate retrieval pipelines (chunking, embeddings, hybrid search, ranking, feedback loops). Own pgvector/Vector DB schemas, latency, relevance and … cost. LLM integration: Integrate OpenAI/Bedrock models, prompt/response orchestration, tool use, guardrails, and evaluation. Data pipelines: Ingest and transform structured/unstructured data; design efficient schemas (Postgres/NoSQL) to support retrieval and analytics. Frontend (React/Next.js): Ship fast, accessible UIs that expose AI features clearly (search, filters, explanations, citations). Architecture More ❯

the AI platform of choice for modern dealmakers. We’re a fast-growing start-up where ideas move quickly from concept to product. Our technology spans backend, frontend, data, and AI, giving our team real scope to shape the future of the product and the company itself. If you’re excited by solving complex problems end-to-end … company’s journey towards achieving its business milestones. Responsibilities (including but not limited to): Backend APIs (Python/FastAPI): Build reliable, secure services that power AI features and data retrieval at scale. RAG & vector search: Design, implement and iterate retrieval pipelines (chunking, embeddings, hybrid search, ranking, feedback loops). Own pgvector/Vector DB schemas, latency, relevance and … cost. LLM integration: Integrate OpenAI/Bedrock models, prompt/response orchestration, tool use, guardrails, and evaluation. Data pipelines: Ingest and transform structured/unstructured data; design efficient schemas (Postgres/NoSQL) to support retrieval and analytics. Frontend (React/Next.js): Ship fast, accessible UIs that expose AI features clearly (search, filters, explanations, citations). Architecture More ❯

Role: Azure Senior Data Engineer Location: Manchester, Lancashire Hybrid working: go into office 1x day a week Salary: £60,000 - £70,000 (dependent on experience) Insight into the Company: A large retail organisation - market leaders in their industry - are looking for a Senior Data Engineer to enter their team. You will be working in a small … team of 3 (including you!) and they are looking to build up the team over the next year. The ideal candidate will have experience with managing data warehouses from start to finish and liaise with IT teams regularly. The role will involve stakeholder management, working with diverse roles, such as software engineers, sales and marketing professionals and more. … Therefore, communicating your projects needs and understanding others is an essential part of this role. Role and Responsibilities: You will design, build and upgrade data pipelines You will work in CI/CD and with Software/DevOps teams in the organisation You will have expertise in azure - from collecting, to transforming to loading! Skills and Experience: Essential More ❯

depending on experience) Employment Type: Full-Time, Permanent Overview A dynamic consultancy is seeking a Senior Analytics Engineer to lead the development of scalable data solutions across a range of high-impact projects. This role offers the opportunity to work at the intersection of data engineering, analytics, and software development, contributing to innovative products and services … that drive real business value. Key Responsibilities Design and implement robust data pipelines using tools such as dbt and Airflow . Collaborate with clients to understand business challenges and deliver tailored data solutions. Transform raw data into structured, actionable datasets for use in AI , predictive modelling , and platform development . Promote engineering best practices … including automation, testing, and code quality. Mentor junior team members and lead technical workstreams to elevate team capability. Essential Skills & Experience Proven experience in analytics engineering , data engineering , or model development roles. Advanced proficiency in Python (including libraries such as NumPy and Pandas) and SQL . Demonstrated ability to deliver production-ready solutions in a commercial environment. Strong More ❯

Data Engineer - PySpark - Palantir - London - £75K Join a dynamic team dedicated to innovative data solutions in Soho! We are seeking a Data Engineer with expertise in PySpark to play a pivotal role as the team and business grow. This permanent, hybrid position offers a unique opportunity to contribute to impactful projects while enjoying a … supportive and collaborative work environment.Key Responsibilities:- Develop and optimise data pipelines using PySpark to ensure efficient data processing- Collaborate with cross-functional teams to deliver high-quality, data-driven insights- Engage in continuous learning and implement best practices in data management- Participate in the deployment and maintenance of data solutions … within the Microsoft ecosystemThe ideal candidate will have strong analytical skills and a passion for data engineering. You will thrive in a setting that values innovation and teamwork. Salary is up to £75,000 plus share options.If you are ready to make a meaningful impact and grow your career, we encourage you to apply for this exciting opportunity More ❯

Job Title: Data Scientist (LLMs & ML) - UK Remote A fast-growing healthcare organization is seeking a Data Scientist with strong experience in machine learning, deep learning, and Large Language Models (LLMs) to help drive innovation and automation in clinical services. This is a full-time, remote UK-based role. ***Unfortunately we can't consider candidates that … solutions (e.g., LangChain, Mastra.ai, Pydantic) for document processing, summarization, and clinical Q&A systems. Develop and optimize predictive models using scikit-learn, PyTorch, TensorFlow, and XGBoost. Design robust data pipelines using tools like Spark and Kafka for real-time and batch processing. Manage ML lifecycle with tools such as Databricks , MLflow , and cloud-native platforms (Azure preferred). … Collaborate with engineering teams to ensure scalable, secure ML infrastructure aligned with compliance standards (e.g., ISO27001). Ensure data governance, particularly around sensitive healthcare data. Share best practices and stay current with developments in AI, ML, and LLMs. Requirements: Proven experience with LLM frameworks and transformer-based architectures. Strong Python skills and familiarity with key ML/DL More ❯

Job title: Data Engineer Client: Award Winning FinTech Firm Rate: Up to £725 p/d Duration: 6-Month Rolling Contract | Long-Term Engagement Location: London Skills: Python/Azure/Databricks/Snowflake The Role: Design and build powerful data pipelines that fuel real-time decisions, machine learning, and next-gen analytics at scale. Transform … raw data into trusted, usable assets - the foundation for everything from executive dashboards to product innovation. Own the data flow from ingestion to insight - using modern tools like Python, Azure, and Databricks. What you need: 7+ years of hands-on experience in data engineering roles. Advanced skills in Python for data pipelines … transformation, and orchestration. Deep understanding of the Azure ecosystem (e.g., Data Factory, Blob Storage, Synapse, etc.) Proficiency in Databricks (or strong equivalent experience with Apache Spark ). Proven ability to work within enterprise-level environments with a focus on clean, scalable, and secure data solutions. If you are the right fit - contact me directly at bbirch More ❯

Job title: Data Engineer Client: Award Winning FinTech Firm Rate: Up to £725 p/d Duration: 6-Month Rolling Contract | Long-Term Engagement Location: London Skills: Python/Azure/Databricks/Snowflake The Role: Design and build powerful data pipelines that fuel real-time decisions, machine learning, and next-gen analytics at scale. Transform … raw data into trusted, usable assets - the foundation for everything from executive dashboards to product innovation. Own the data flow from ingestion to insight - using modern tools like Python, Azure, and Databricks. What you need: 7+ years of hands-on experience in data engineering roles. Advanced skills in Python for data pipelines … transformation, and orchestration. Deep understanding of the Azure ecosystem (e.g., Data Factory, Blob Storage, Synapse, etc.) Proficiency in Databricks (or strong equivalent experience with Apache Spark ). Proven ability to work within enterprise-level environments with a focus on clean, scalable, and secure data solutions. If you are the right fit - contact me directly at bbirch More ❯

Job title: Data Engineer Client: Award Winning FinTech Firm Rate: Up to £725 p/d Duration: 6-Month Rolling Contract | Long-Term Engagement Location: London Skills: Python/Azure/Databricks/Snowflake The Role: Design and build powerful data pipelines that fuel real-time decisions, machine learning, and next-gen analytics at scale. Transform … raw data into trusted, usable assets - the foundation for everything from executive dashboards to product innovation. Own the data flow from ingestion to insight - using modern tools like Python, Azure, and Databricks. What you need: 7+ years of hands-on experience in data engineering roles. Advanced skills in Python for data pipelines … transformation, and orchestration. Deep understanding of the Azure ecosystem (e.g., Data Factory, Blob Storage, Synapse, etc.) Proficiency in Databricks (or strong equivalent experience with Apache Spark ). Proven ability to work within enterprise-level environments with a focus on clean, scalable, and secure data solutions. If you are the right fit - contact me directly at bbirch More ❯

New Role: Data Science Engineer (LLM Focus) Salary: £55,000-£75,000 + share options Hybrid/Remote: Flexible but with regular London meetings? Location: UK-based (preferably London) We're looking for a Data Science Engineer to join a fast-moving team working on NLP pipelines and data at scale. This is a … great opportunity for someone passionate about impactful data work and modern AI tools. What We're Looking For: 1+ years' experience in data science or engineering.? Strong Python skills and experience with LLMs/NLP.? Experience building scalable data pipelines.? Bonus: GCP, Docker, geospatial data, Postgres/PostGIS If you're a … hands-on data scientist/engineer looking for your next step, we'd love to hear from you. Interested? Apply now! Oscar Associates (UK) Limited is acting as an Employment Agency in relation to this vacancy. To understand more about what we do with your data please review our privacy policy in the privacy section of More ❯

making process, with the founders having had firsthand experience of the lengthiness and challenges of deal processes in the past. Their stack spans across back end, front end, data and crucially AI.This is a great opportunity for an entrepreneurial software engineer who wants to play a part in shaping the technical vision of this business and work on … their product from an early stage. What you'll work on: Backend APIs (Python/FastAPI): Build and maintain secure, high-performance services that drive AI features and data access at scale. RAG & vector search: Design and improve retrieval pipelines (embeddings, chunking, hybrid search, ranking, feedback loops), owning schema design, latency, and relevance across vector databases. LLM integration … Connect and orchestrate large language models (OpenAI, Bedrock, etc.), manage prompts, tools, safeguards, and evaluation. Data pipelines: Ingest, clean, and transform structured and unstructured data; design efficient schemas (Postgres/NoSQL) for search and analytics. Frontend (React/Next.js): Deliver user-friendly, performant UIs that make AI-powered features (search, filters, explanations, citations) clear and accessible. More ❯

Overview An excellent opportunity for a talented and motivated Product Manager to guide and support the delivery of the set of Science Data Processing (SDP) pipelines for the Mid and Low SKA Telescopes. The role is internally known as SKA Operations Scientist/SAFe Product Manager. At the SKA Observatory, Product Managers … are at the heart of turning vision into reality. In this role, you will support engineering and development teams as they deliver the Science Data Processor (SDP) pipeline software for both the Mid and Low telescopes. Your job is to help teams ask the right questions and solve the right problems. The goal is simple: ensure that … product goals, impacts, and deliverables. Ensure that the pipelines integrate effectively with telescope systems, data archives, and international science data networks. Drive continuous improvement in pipeline reliability, usability, and scalability to meet the challenges of petabyte-scale data processing. Represent the product to internal and external stakeholders, providing clear communication of progress, risks More ❯

Data & Insight Manager - Ecommerce Job Title: Data & Insight Manager Salary Range: £60,000 - £75,000 + discretionary bonus Location: Cardiff Area, Wales (Hybrid - 2 days/week in office) (With some potential flexibility) Our client is a global consumer brand with nearly a century of heritage and a reputation for innovation. Following a recent acquisition by … team and high-growth ambitions, this is a rare chance to join at a foundational stage, helping professionalise a business that's scaling fast across international markets. The Data & Insight Manager will be the first dedicated analytics hire in the business — owning everything from technical setup to commercial storytelling. You'll work cross-functionally across eCommerce, marketing, and … operations to unlock the value of data and drive critical decision-making. This is a hands-on, high-impact role for someone who thrives on solving problems, building from scratch, and creating business value through data. Essential Skills: Strong SQL and experience with BI tools (e.g. Looker, Tableau, Data Studio) Proven experience building and maintaining data More ❯

Data & Insight Manager - Ecommerce Job Title: Data & Insight Manager Salary Range: £60,000 - £75,000 + discretionary bonus Location: Cardiff Area, Wales (Hybrid - 2 days/week in office) (With some potential flexibility) Our client is a global consumer brand with nearly a century of heritage and a reputation for innovation. Following a recent acquisition by … team and high-growth ambitions, this is a rare chance to join at a foundational stage, helping professionalise a business that's scaling fast across international markets. The Data & Insight Manager will be the first dedicated analytics hire in the business — owning everything from technical setup to commercial storytelling. You'll work cross-functionally across eCommerce, marketing, and … operations to unlock the value of data and drive critical decision-making. This is a hands-on, high-impact role for someone who thrives on solving problems, building from scratch, and creating business value through data. Essential Skills: Strong SQL and experience with BI tools (e.g. Looker, Tableau, Data Studio) Proven experience building and maintaining data More ❯

Job title: Bi Analyst - Retention Data Location: London, SW1E 5JL (on site 3 days per week) Contract length: 12 months Pay rate: £200.00 per day (PAYE) As we rapidly expand our ecosystem of products and services, we are looking for a skilled analyst to join our business to work across several high-profile projects focused on customer retention. … work with large databases and cutting-edge tech stacks - you will use SQL/Python and BI tools (such as Tableau/Qlik) to mine, synthesize, and visualise data and influence work across a range of stakeholders. Responsibilities Support the analytics needs of the business by building automated scalable reporting solutions focused on customer retention. Create holistic dashboards … by combining data from both internal and external data sources for presentations to senior business leaders regarding customer retention. Work closely with mission teams to identify gaps in the data capture strategy related to customer behaviour and collaboratively implement enhancements. Go one step further and provide actionable insights and recommendations related to key customer More ❯

Job title: Bi Analyst - Retention Data Location: London, SW1E 5JL (on site 3 days per week) Contract length: 12 months Pay rate: £200.00 per day (PAYE) As we rapidly expand our ecosystem of products and services, we are looking for a skilled analyst to join our business to work across several high-profile projects focused on customer retention. … work with large databases and cutting-edge tech stacks - you will use SQL/Python and BI tools (such as Tableau/Qlik) to mine, synthesize, and visualise data and influence work across a range of stakeholders. Responsibilities Support the analytics needs of the business by building automated scalable reporting solutions focused on customer retention. Create holistic dashboards … by combining data from both internal and external data sources for presentations to senior business leaders regarding customer retention. Work closely with mission teams to identify gaps in the data capture strategy related to customer behaviour and collaboratively implement enhancements. Go one step further and provide actionable insights and recommendations related to key customer More ❯

About The Role We are looking for an experienced Test Manager with a strong background in data platform testing, particularly across Microsoft Fabric, Azure services, PySpark notebooks, and automated testing frameworks. You will play a pivotal role in ensuring quality and governance across data pipelines, lakehouses, and reporting layers. This role requires a hands-on leader … . Lead and manage the QA effort across multiple agile teams. Drive the development and maintenance of automated testing for: 1-PySpark notebooks in Fabric/Databricks 2-Data pipelines and transformations 3-Delta tables and lakehouse validation Embed testing into CI/CD pipelines (Azure DevOps or GitHub Actions). Ensure quality gates are enforced through all … stages of the medallion architecture (bronze, silver, gold). Define test approaches for schema validation, data reconciliation, and business rule testing. Collaborate with data engineers, architects, product owners, and DevOps to deliver high-quality, scalable solutions. Establish reusable patterns for parameterized notebook testing and data-driven assertions. Own test planning, execution, environment readiness, and More ❯

is unique. We'd love you to join us. Delivery. Work. Projects. As a Digital Performance Analyst in DWP Digital's Specialist Devices team, you'll help turn data into decisions that improve the tools and services our colleagues rely on every day. You'll be part of a user-focused team that owns the end-to-end … ll collaborate with product owners, engineers, and designers to continuously improve services based on evidence and user needs. You'll also support wider government priorities around digital inclusion, data transparency, and service efficiency. It's a great opportunity to work at the heart of digital transformation in the UK's largest government department, where your analysis will help … to build insightful dashboards and reports that drive decision-making. SQL Proficiency - You're confident writing custom SQL to perform calculations, join multiple datasets, and feed clean, structured data into dashboards and analysis tools. Digital Workplace Insight - You've worked in digital workplace environments and know how to analyse data from digital systems to surface trends More ❯

demonstrating the power of our platform and ensuring our clients' success. Key Responsibilities Client Engagement & Discovery: Collaborate closely with enterprise clients to understand their business processes, pain points, data ecosystems, and strategic goals. Solution Design & Architecture: Design end-to-end solutions by mapping client requirements to our platform's offerings (e.g., Knowledge Hubs, Conversational AI, Real-time Insights … and Agentic Automation). Create detailed architecture documents, data flow diagrams, and technical specifications. MCP Workflow & Ontology Design: Architect the core intelligence for client solutions. This includes designing client-specific ontologies and defining the structure, logic, and composition of Model Context Protocols (MCPs) to automate complex reasoning and workflows. Technical Leadership: Serve as the deep technical expert on … the platform for both clients and internal teams. Provide guidance on best practices for data integration, RAG implementation, and agentic automation. Proof of Concept & Prototyping: Lead the technical design and development of Proof of Concept (PoC) projects to demonstrate the feasibility and value of our solutions for specific client use cases. Cross-functional Collaboration: Work closely with our More ❯

scheme Internal coaching/mentoring system throughout your time here Focus on training and career progression Family friendly policies Happy to talk about flexible working The Opportunity The Data Analyst is the subject matter expert for a set of datasets that underpin Landmark’s many products and services. You will be responsible for ensuring the quality and currency … migration. In addition, the team also gets involved in project work where you may be asked to contribute based on your technical or domain expertise. About You The Data Analyst will be inquisitive with a desire to understand and resolve problems. You will also be an effective communicator, with the ability to plan, allocate and manage work for … yourself and junior team members. You will also have/be: A qualification in GIS or Data related discipline or equivalent professional experience Practical experience of working in a data analysis or data curation role Practical experience of authoring ETL/ELT processes following current best data governance practises, of problem-solving More ❯

pipelines and AI infrastructure. Develop platform-level test solutions and automation frameworks using Python, Terraform, and modern cloud-native practices. Contribute to the platform’s CI/CD pipeline by integrating automated testing, resilience checks, and observability hooks at every stage. Lead initiatives that drive testability, platform resilience, and validation as code across all layers of the ML … provision cloud infrastructure. Strong understanding of software engineering best practices: code quality, reliability, performance optimization, and observability. Preferred Qualifications Exposure to machine learning workflows, model lifecycle management, or data engineering platforms. Experience with distributed systems, event-driven architectures (e.g., Kafka), and big data platforms (e.g., Spark, Databricks). Familiarity with banking or financial domain use cases … including data governance and compliance-focused development. Knowledge of platform security, monitoring, and resilient architecture patterns. More ❯

Data Engineer - Generative AI Projects ?? Hybrid | 2-3 Days On-Site | Up to £55,000 + Bonus + Package ?? British Citizens Only | BPSS Clearance Required We're hiring a Data Engineer with a passion for automation, clean pipelines, and the future of AI. This is your chance to shape how data powers Generative AI … on innovative internal use-cases - from AI assistants to intelligent search and document automation - this is the perfect next step. ?? What You'll Be Doing Building and maintaining data pipelines across structured & unstructured sources Collaborating with internal teams to support Generative AI and NLP projects Ensuring data is secure, compliant, high-quality, and easy to access … customers across multiple teams and functions ?? What You'll Bring Experience with SQL & NoSQL databases (e.g. MS SQL, MongoDB, Neo4J) Python skills for scripting and automation ETL and data exchange experience (e.g. APIs, ESB tools) Knowledge of Big Data (e.g. Hadoop) Curiosity about AI, particularly NLP, OCR or Generative AI ?? Bonus Points For Docker/containerisation More ❯

web applications, focusing on lead generation workflows. Optimize and scale platform performance to handle increasing lead volumes efficiently. Integrate and manage APIs for customer eligibility checks, ensuring encrypted data transmission and receipt in compliance with GDPR and FCA guidelines. Collaborate with marketing teams and partners to enhance lead qualification processes using automated systems. Build responsive, conversion-driven user … interfaces with a seamless customer journey. Ensure platform security, including data encryption and compliance with UK regulatory standards. Troubleshoot, debug, and upgrade existing systems to improve functionality and user experience. Work with cloud infrastructure (AWS or Replit) to deploy, manage, and scale applications. Participate in code reviews, testing, and documentation to maintain high-quality standards. Stay updated on … and Express.js for server-side logic. Familiarity with Formik and Yup for form handling and validation. Database and Storage : Experience with Supabase or similar PostgreSQL-based solutions for data management. Ability to integrate and manage relational databases securely. API Integration : Strong experience in working with RESTful APIs (e.g., Numverify, ZeroBounce) to send and receive encrypted data. Knowledge of More ❯

plus 28.9% employer pension contributions, hybrid working with 60% home working, flexible hours, and great work life balance. Would you like to kickstart a career in data and technology while gaining a recognised qualification in a supportive, diverse environment? DWP. Digital with Purpose. We're offering a fantastic opportunity to join our Level 5 Data Engineering … UK's largest and most impactful digital organisations. At DWP Digital, we design and build the technology behind vital services like Universal Credit, pensions, and child maintenance. Our data engineers play a key role in making sure these services are powered by reliable, high-quality data, helping teams make better decisions and improving outcomes for millions … of people. As an apprentice, you'll join our Data Platforms and Operations team , working with cloud technologies like AWS and tools such as Informatica. You'll learn how to build and maintain data pipelines, support platform migrations, and help deliver data products that are secure, efficient, and user-focused. You'll be supported More ❯

plus 28.9% employer pension contributions, hybrid working with 60% home working, flexible hours, and great work life balance. Would you like to kickstart a career in data and technology while gaining a recognised qualification in a supportive, diverse environment? DWP. Digital with Purpose. We're offering a fantastic opportunity to join our Level 5 Data Engineering … UK's largest and most impactful digital organisations. At DWP Digital, we design and build the technology behind vital services like Universal Credit, pensions, and child maintenance. Our data engineers play a key role in making sure these services are powered by reliable, high-quality data, helping teams make better decisions and improving outcomes for millions … of people. As an apprentice, you'll join our Data Platforms and Operations team , working with cloud technologies like AWS and tools such as Informatica. You'll learn how to build and maintain data pipelines, support platform migrations, and help deliver data products that are secure, efficient, and user-focused. You'll be supported More ❯