301 to 325 of 399 Data Pipeline Jobs in England

the DNA of the platform What We’re Looking For Proven track record designing or architecting distributed, high-volume systems Experience with event-driven architectures and high-throughput data pipelines (Kafka) Strong knowledge of at least one of: C#, .NET, Golang, or TypeScript , plus SQL/databases Ability to balance hands-on technical leadership with high-level solution More ❯

problem that you're solving, and you will have the opportunity to shape the direction of the product. You will evaluate your own work, leveraging our data pipeline and frameworks that we have established to understand the impact your features have on our commercial objectives and pivoting where necessary. You will consider the long-term direction of More ❯

problem that you're solving, and you will have the opportunity to shape the direction of the product. You will evaluate your own work, leveraging our data pipeline and frameworks that we have established to understand the impact your features have on our commercial objectives and pivoting where necessary. You will consider the long-term direction of More ❯

Job Title: Lead Data Engineer Location: Warwick (Hybrid - once per week onsite) Remuneration: From £700 per day (via umbrella company) Contract Details: Fixed Term Contract, 6 months, Full Time Are you ready to take the lead in transforming data engineering? Our client, a forward-thinking organisation, is seeking a highly skilled and hands-on Lead Data Engineer to drive their Data & Insights platform built on the Azure stack. This is a Databricks-heavy role , ideal for someone who thrives in technical environments and enjoys working directly with cutting-edge tools. Why This Role Stands Out This is not just a leadership role - it's a hands … on engineering opportunity where your Databricks expertise will be front and centre . You'll be the go-to person for all things Databricks, from architecture and configuration to pipeline promotion and troubleshooting. Key Responsibilities Lead and deliver hands-on data engineering across all layers of the Azure-based data platform. Act as the Databricks More ❯

This team leverages technical expertise in actuarial modelling, statistical analysis, and risk management solutions to support a high-impact team of 90+ underwriters. You will build and maintain data pipelines and data models to elevate the Analytics team and help answer the most difficult business challenges. Key requirements Experience in the London or Lloyd's insurance … markets is essential Have an Advanced degree in Statistics, Computer Science, Mathematics or a related field or relevant industry industry experience Have knowledge of programming and data manipulation skills such as SQL and Python, and version control (Github) is essential Experience building multi-step ETL jobs/data pipelines and working with job scheduling systems is … a plus Demonstrated ability to thrive with minimal oversight and process. You connect the dots across business needs and independently drive projects forward Have strong data visualization skills and experience with Power BI is a bonus Have experience with cloud technologies, preferably Microsoft Azure More ❯

Overview Role Description In this role you will collaborate with data scientists, ML engineers, and analysts to design and implement solutions that extract insights from unstructured text. Projects may include topic modelling and entity recognition, text generation, ontology creation, and conversational AI. This role demands strong technical expertise in NLP and a passion for innovation and problem-solving. … level ML/GenAI strategies aligned with business objectives. Designing and developing NLP solutions using techniques such as text classification, NER, topic modeling, and text generation. Collaborating with data engineers and MLOps teams to ensure clean data pipelines and seamless model integration. Applying LLMs (e.g., GPT, BERT, LLaMA2) and knowledge graphs to enhance natural language understanding … and reasoning. Conducting exploratory data analysis and developing robust pre-processing strategies for unstructured text. Evaluating NLP models, algorithms, and architectures to recommend optimal solutions. Communicating insights through reports, visualizations, and presentations for both technical and non-technical audiences. Mentoring junior team members and staying current with advances in NLP, LLMs, and GenAI. Essential Skills A Bachelor's More ❯

/09/2025 About the Role As an AI Apprentice, you will have the unique opportunity to learn directly from our team of AI researchers, engineers, and data scientists. This role is designed to provide structured, real-world experience in machine learning, data processing, and model deployment while working on exciting projects. You'll gain … practical experience in various stages of AI development, from data preprocessing and algorithm implementation to testing and optimizing AI models. Key Responsibilities Assist with data collection, preprocessing, and cleaning tasks. Support the development and training of machine learning models. Collaborate on AI research projects under the guidance of senior AI engineers. Perform exploratory data … Participate in regular team meetings and brainstorming sessions. Gain experience with popular AI/ML tools and frameworks such as Python, TensorFlow, PyTorch, and Scikit-learn. Learn about data pipelines, from data ingestion to model deployment. Deepen your understanding of algorithms, AI ethics, and the latest trends in AI. Hands-on experience with real-world AI More ❯

ll join the Platform & AI Enablement team within our GPTO Engineering organization, reporting to the Sr. Director Engineering API. The team is responsible for building and supporting the data and AI platforms that underpin our industry-leading Business Planning Software solutions. This is a highly hands-on role focused on designing, building, and operating scalable systems that integrate … event-driven, batch & streaming data flows, and AI capabilities into the platform. Your Impact Build and maintain core platform capabilities that support high-throughput batch, streaming, and AI-powered workloads. Develop resilient, observable, and scalable systems using Apache Kafka, Flink, Pulsar, and cloud-native tools. Collaborate with AI/ML engineers to operationalize models and enable generative AI … use cases such as prompt-based insights or automation. Deliver reliable integrations with data lakes, event stores, and analytics systems, ensuring data flows efficiently across the business planning ecosystem. Contribute full-stack code where needed, including React-based frontends and backend services in Java and Python. Write clean, maintainable, well-tested code with an emphasis on More ❯

Job Title: Lead Data Engineer Industry: Banking and Financial Services Location : Flexible- Basingstoke (3 Days a Week) Overview of Role: This role will lead the design and optimisation of our Azure Databricks data platform , ensuring it is secure, scalable, and well governed. You'll take ownership of change controls, lineage, and documentation while driving adoption of … best practice. Alongside this, you'll support AI/ML initiatives and provide technical guidance to colleagues, driving value from data. Key Responsibilities Lead the roadmap for data engineering, ensuring alignment with business needs and regulatory requirements. Own the design and optimisation of the Azure Databricks platform, focusing on performance, resilience, and compliance. Implement change controls, lineage, and … observability to detect and resolve issues. Maintain clear documentation of data pipelines, models, and governance processes. Implement tools to improve data quality, efficiency, and adoption. Support AI/ML delivery by building/optimising, ensuring platform readiness. Manage SQL workloads within Databricks, balancing cost and performance Ensure access controls, GDPR compliance, and security standards Work with More ❯

building the AI platform of choice for modern dealmakers. Were a fast-growing start-up where ideas move quickly from concept to product. Our technology spans backend, frontend, data, and AI, giving our team real scope to shape the future of the product and the company itself. If youre excited by solving complex problems end-to-end, working … the companys journey towards achieving its business milestones. Responsibilities (including but not limited to): Backend APIs (Python/FastAPI): Build reliable, secure services that power AI features and data retrieval at scale. RAG & vector search: Design, implement and iterate retrieval pipelines (chunking, embeddings, hybrid search, ranking, feedback loops). Own pgvector/Vector DB schemas, latency, relevance and … cost. LLM integration: Integrate OpenAI/Bedrock models, prompt/response orchestration, tool use, guardrails, and evaluation. Data pipelines: Ingest and transform structured/unstructured data; design efficient schemas (Postgres/NoSQL) to support retrieval and analytics. Frontend (React/Next.js): Ship fast, accessible UIs that expose AI features clearly (search, filters, explanations, citations). Architecture More ❯

the AI platform of choice for modern dealmakers. We’re a fast-growing start-up where ideas move quickly from concept to product. Our technology spans backend, frontend, data, and AI, giving our team real scope to shape the future of the product and the company itself. If you’re excited by solving complex problems end-to-end … company’s journey towards achieving its business milestones. Responsibilities (including but not limited to): Backend APIs (Python/FastAPI): Build reliable, secure services that power AI features and data retrieval at scale. RAG & vector search: Design, implement and iterate retrieval pipelines (chunking, embeddings, hybrid search, ranking, feedback loops). Own pgvector/Vector DB schemas, latency, relevance and … cost. LLM integration: Integrate OpenAI/Bedrock models, prompt/response orchestration, tool use, guardrails, and evaluation. Data pipelines: Ingest and transform structured/unstructured data; design efficient schemas (Postgres/NoSQL) to support retrieval and analytics. Frontend (React/Next.js): Ship fast, accessible UIs that expose AI features clearly (search, filters, explanations, citations). Architecture More ❯

the development and implementation of robust Lab/digital infrastructure within our R&D laboratories for the UK and US locations. This is pivotal in enabling efficient, automated data exchange between laboratory instruments and enterprise informatics systems such as Electronic Lab notebooks (ELN)s, Laboratory Information Management Systems (LIMS) and other relevant data setups. The ideal … implement, and maintain secure, scalable IT/OT interfaces between lab instruments (e.g., bioreactors, plate readers, liquid handlers) and informatics systems (ELN, LIMS, SDMS, etc.) Collaborate with IT, Data science and scientific teams in R&D to identify integration needs and define architecture for real-time or scheduled automated data transfer. Develop or configure middleware, APIs … and custom connectorseither in-house or in collaboration with external vendorsto enable automated data capture, contextualization, and storage. Manage instrument network connectivity (e.g., OPC-UA, MQTT, REST APIs, TCP/IP) and data pipelines from edge devices to cloud/on-premise environments. Support system validation efforts (e.g., CSV, 21 CFR Part 11 compliance) in partnership More ❯

Role: Azure Senior Data Engineer Location: Manchester, Lancashire Hybrid working: go into office 1x day a week Salary: £60,000 - £70,000 (dependent on experience) Insight into the Company: A large retail organisation - market leaders in their industry - are looking for a Senior Data Engineer to enter their team. You will be working in a small … team of 3 (including you!) and they are looking to build up the team over the next year. The ideal candidate will have experience with managing data warehouses from start to finish and liaise with IT teams regularly. The role will involve stakeholder management, working with diverse roles, such as software engineers, sales and marketing professionals and more. … Therefore, communicating your projects needs and understanding others is an essential part of this role. Role and Responsibilities: You will design, build and upgrade data pipelines You will work in CI/CD and with Software/DevOps teams in the organisation You will have expertise in azure - from collecting, to transforming to loading! Skills and Experience: Essential More ❯

leadership, driving integration initiatives in a complex and evolving environment. Key Responsibilities Scope, design, and build integrations across third-party platforms leveraging AWS infrastructure Develop and maintain scalable data pipelines using Python, ensuring data quality, consistency, and reliability Ensure secure and efficient data exchange between platforms by applying best-practice design principles Collaborate with … Data and Infrastructure teams to translate requirements into robust technical solutions that align with the overall integration architecture Produce and maintain clear documentation of integration flows Required Skills & Experience Strong experience with RESTful API integrations Proven understanding of integration ETL/ELT concepts Advanced proficiency in Python for data transformation, scripting, and automation, including handling sparsely … populated data sets Experience with database structures (SQL/DDL), including the creation and management of staging tables Ability to translate requirements into actionable technical tasks Excellent problem-solving and communication skills Solid understanding of software engineering principles (testing, version control, CI/CD) Nice to Have Experience with AWS services such as Glue, Aurora, S3, and RDS More ❯

depending on experience) Employment Type: Full-Time, Permanent Overview A dynamic consultancy is seeking a Senior Analytics Engineer to lead the development of scalable data solutions across a range of high-impact projects. This role offers the opportunity to work at the intersection of data engineering, analytics, and software development, contributing to innovative products and services … that drive real business value. Key Responsibilities Design and implement robust data pipelines using tools such as dbt and Airflow . Collaborate with clients to understand business challenges and deliver tailored data solutions. Transform raw data into structured, actionable datasets for use in AI , predictive modelling , and platform development . Promote engineering best practices … including automation, testing, and code quality. Mentor junior team members and lead technical workstreams to elevate team capability. Essential Skills & Experience Proven experience in analytics engineering , data engineering , or model development roles. Advanced proficiency in Python (including libraries such as NumPy and Pandas) and SQL . Demonstrated ability to deliver production-ready solutions in a commercial environment. Strong More ❯

depending on experience) Employment Type: Full-Time, Permanent Overview A dynamic consultancy is seeking a Senior Analytics Engineer to lead the development of scalable data solutions across a range of high-impact projects. This role offers the opportunity to work at the intersection of data engineering, analytics, and software development, contributing to innovative products and services … that drive real business value. Key Responsibilities Design and implement robust data pipelines using tools such as dbt and Airflow . Collaborate with clients to understand business challenges and deliver tailored data solutions. Transform raw data into structured, actionable datasets for use in AI , predictive modelling , and platform development . Promote engineering best practices … including automation, testing, and code quality. Mentor junior team members and lead technical workstreams to elevate team capability. Essential Skills & Experience Proven experience in analytics engineering , data engineering , or model development roles. Advanced proficiency in Python (including libraries such as NumPy and Pandas) and SQL . Demonstrated ability to deliver production-ready solutions in a commercial environment. Strong More ❯

hands-on role to deliver applications working with other staff and partners. Collaborate with cross-functional teams to design, develop, and implement scalable and reliable cloud applications. Utilize data science and analytics to enhance application functionality and performance. Work with the data team to create and deploy machine learning models and AI-driven solutions for real … development/integration. Familiarity with enterprise architecture design and delivery (Azure, GitHub, etc). Proven experience in building cloud applications using Microsoft Azure/cloud. Strong interest in data science and analytics. Excellent problem-solving skills and attention to detail. Knowledge of effective business analysis - ability to gather, document, and analyse business requirements effectively and the experience creating … solve Preferred Qualifications: Experience in manufacturing, retail or hospitality industries. Knowledge of other programming languages and frameworks. Interest and experience with AI solution engineering Interest and experience of data analytics and data pipelines. BENEFITS BAKED IN Free food and drink when working 50% off food and drink when not working 25 days holiday + Bank holiday More ❯

iO Associates are partnering with a leading consultancy to support NHS England in the design and delivery of innovative data solutions aimed at transforming healthcare finance and operational decision-making. As part of the Data and Analytics Engineering team, you will play a key role in developing and deploying data products within Palantir Foundry. … within Foundry, mentor junior team members, and serve as a key point of contact for NHS England stakeholders. This is a unique opportunity to apply your expertise in data engineering, ETL pipelines, and analytics to solve complex challenges within one of the UK's most impactful public sector programmes. Responsibilities Lead the design, development, and deployment of data-driven solutions within Palantir Foundry Build and manage ETL pipelines and data workflows to integrate large and complex NHS datasets Provide day-to-day technical leadership, mentoring junior developers and guiding the wider team Act as a trusted advisor to consultancy and NHS England stakeholders, ensuring solutions meet business needs and governance requirements Champion delivery methodologies More ❯

Job Title: Machine Learning Engineer (Data/ML) Location: London/Hybrid - Once a week (Thursday) in London Duration: 6 Months Day Rate: £500 per day overall assignment rate to umbrella. This is a greenfield based project with crossover from another tech team conducting a similar model. You will collaborate with the Data Scientist, Data … cross-functional Tech Teams , supporting the build phase while contributing hands-on technical expertise. Key Responsibilities: Work with the Senior Engineer/Tech Lead to implement and optimise data pipelines and ML models. Develop high-quality, maintainable code in Python and PySpark. Use SQL and Jupyter to analyse, transform and validate data for ML use cases. … technical teams to deliver solutions that support business growth. Skills & Experience Required: Strong hands-on experience in Python, PySpark, SQL, Jupyter . Experience in Machine Learning engineering or data-focused development. Exposure to working in cloud platforms (AWS/Azure) . Ability to collaborate effectively with senior engineers, data scientists, and architects. Proactive, detail-oriented, and More ❯

New Role: Data Science Engineer (LLM Focus) ?? Salary: £55,000-£75,000 + share options ?? Hybrid/Remote: Flexible but with regular London meetings ?? Location: UK-based (preferably London) We're looking for a Data Science Engineer to join a fast-moving team working on NLP pipelines and data at scale. This is a … great opportunity for someone passionate about impactful data work and modern AI tools. What We're Looking For: ? 1+ years' experience in data science or engineering. ? Strong Python skills and experience with LLMs/NLP. ? Experience building scalable data pipelines. ? Bonus: GCP, Docker, geospatial data, Postgres/PostGIS If you're a … hands-on data scientist/engineer looking for your next step, we'd love to hear from you. ?? Interested? Apply now! Oscar Associates (UK) Limited is acting as an Employment Agency in relation to this vacancy. To understand more about what we do with your data please review our privacy policy in the privacy section of More ❯

London, United Kingdom Posted on 12/09/2025 We're hiring a Data Scientist with strong Generative-AI experience to design, build, and ship AI-powered tools end-to-end. You'll work in a small, multi-disciplinary team and take ownership from discovery to deployment: scoping use-cases, building prototypes, hardening them for production, and … cost; add guardrails and red-teaming. Productionise : package as services/APIs or lightweight apps (e.g., Streamlit/Gradio/React), containerise, and integrate via CI/CD. Data pipelines : design chunking/embedding strategies, pick vector stores, manage prompt/versioning, and monitor drift & quality. Model strategy : select and mix providers (hosted and open-source), fine-tune … sensible, and optimise for cost/perf/privacy. Stakeholder enablement : translate problems into measurable KPIs, run discovery, document clearly, and hand over maintainable solutions. Good practice : apply data ethics, security and privacy by design; align to service standards and accessibility where relevant. Tech you'll likely use LLM frameworks : LangChain, LlamaIndex (or similar) Cloud & Dev : Azure/ More ❯

cloud native technologies (e.g. AWS, GCP, Snowflake, Databricks) under the guidance of senior team members Key Skills and Experience required - A strong academic background in Computer Science, Engineering, Data Science, or a related technical discipline Proficiency in Java or Python Strong written and verbal communication skills Proactive learner with the ability to self-teach new tools and technologies … an ability to educate others Familiarity with the Linux command line and shell scripting Understanding of basic data concepts - file formats (e.g. CSV, Parquet), data pipelines, and storage layers Exposure to containers and Podman or Docker Comfortable using Git; an awareness of CI/CD practices and tools such as GitHub Actions or Azure DevOps Nice … development workflows using e.g., VSCode Basic understanding of cloud platforms like AWS or GCP Experience contributing to open-source or internal code templates, demos, or accelerators Familiarity with data catalog services (e.g. Hive, Polaris, Glue) This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required. More ❯

making process, with the founders having had firsthand experience of the lengthiness and challenges of deal processes in the past. Their stack spans across back end, front end, data and crucially AI.This is a great opportunity for an entrepreneurial software engineer who wants to play a part in shaping the technical vision of this business and work on … their product from an early stage. What you'll work on: Backend APIs (Python/FastAPI): Build and maintain secure, high-performance services that drive AI features and data access at scale. RAG & vector search: Design and improve retrieval pipelines (embeddings, chunking, hybrid search, ranking, feedback loops), owning schema design, latency, and relevance across vector databases. LLM integration … Connect and orchestrate large language models (OpenAI, Bedrock, etc.), manage prompts, tools, safeguards, and evaluation. Data pipelines: Ingest, clean, and transform structured and unstructured data; design efficient schemas (Postgres/NoSQL) for search and analytics. Frontend (React/Next.js): Deliver user-friendly, performant UIs that make AI-powered features (search, filters, explanations, citations) clear and accessible. More ❯

detection logic or supporting telemetry. Skills and Attributes Detection Engineering Framework experience Threat Hunting & Log Analysis experience Threat Modelling & Framework Alignment experience Desirable (but not essential) Data Pipeline & Telemetry Management Benefits At Claranet, we go the extra mile with our people—because we believe in building a workplace where everyone feels valued and supported. Our flexible benefits More ❯

Overview An excellent opportunity for a talented and motivated Product Manager to guide and support the delivery of the set of Science Data Processing (SDP) pipelines for the Mid and Low SKA Telescopes. The role is internally known as SKA Operations Scientist/SAFe Product Manager. At the SKA Observatory, Product Managers … are at the heart of turning vision into reality. In this role, you will support engineering and development teams as they deliver the Science Data Processor (SDP) pipeline software for both the Mid and Low telescopes. Your job is to help teams ask the right questions and solve the right problems. The goal is simple: ensure that … product goals, impacts, and deliverables. Ensure that the pipelines integrate effectively with telescope systems, data archives, and international science data networks. Drive continuous improvement in pipeline reliability, usability, and scalability to meet the challenges of petabyte-scale data processing. Represent the product to internal and external stakeholders, providing clear communication of progress, risks More ❯