626 to 650 of 802 Data Pipeline Jobs in England

on nurturing Al talent, fostering knowledge sharing, and continuously evolving Al practices across the organization Location: Chennai, Onsite Experience: • 11+ years of experience in Al, machine learning, or data science , with a proven track record of delivering Al solutions. • 7+ years of experience in a leadership or architecture role, ideally with some experience in leading a Centre of … multiple industries is advantageous (e.g., healthcare, finance, retail). Skills: AI/ML Expertise: Strong understanding of machine learning algorithms, deep learning, natural language processing, computer vision, and data-driven problem-solving techniques. Architecture Skills: Proven ability to design and architect scalable, reliable, and high-performance Al solutions. Leadership and Communication: Excellent leadership skills with the ability to … in solution design, integration, and deployment. Consulting and Advisory: Work closely with stakeholders to identify business requirements and translate them into Al-powered solutions, including machine learning models, data pipelines, and Al- driven processes. Platform Selection and Integration: Evaluate and select appropriate Al tools, platforms, and technologies to meet business goals. Oversee integration with existing systems, ensuring scalability More ❯

develop, and deploy end-to-end AI/ML-powered features and applications using Python and modern frameworks. Build, integrate, and maintain machine learning models and solutions-from data pipelines to API deployment and user-facing interfaces. Collaborate with product managers, compliance experts, and engineers to translate business needs into intelligent technologies. Develop robust backend systems and APIs … models (LLMs, predictive/classification, agentic workflows), ensuring security and compliance standards are met. Implement best practices for MLOps, including CI/CD, automated testing, model versioning, and data validation. Architect integrations with vector databases, cloud platforms, and retrieval-augmented generation systems. Qualifications: 3+ years of experience delivering full-stack AI/ML applications in production. Proficiency in More ❯

services using .NET Core/.NET 6+ Collaborate efficiently on an AI-driven project alongside the customers Lead Backend Engineer Work closely with ML Engineers, App Developers, and Data teams for integrated feature delivery Design and build RESTful APIs to expose AI models and LLM prompt-based services Integrate trained ML models into backend systems supporting real product … features Build and maintain infrastructure for data ingestion, processing, and secure storage Support full end-to-end testing with QA and development teams to prevent security or data issues Create services to manage data ingestion for model training and dynamic personalization Implement access controls, password policies, and SSO using existing client authentication models Secure … APIs and data pipelines with encryption and appropriate authentication measures Required Skills and Experience Proven experience in .NET Core/.NET 6+ development Strong understanding of REST API design and backend microservices architecture Experience working on AI/ML integrations in production environments Familiarity with secure coding practices and enterprise-level data protection standards Comfortable with More ❯

via Docker/Kubernetes and integrate with orchestration systems (e.g., Airflow, custom schedulers). ? Work with platform engineers to embed Spark jobs into InfoSum's platform APIs and data pipelines. ? Troubleshoot job failures, memory and resource issues, and execution anomalies across various runtime environments. ? Optimize Spark job performance and advise on best practices to reduce cloud compute and … in at least two major cloud environments (AWS, GCP, Azure). ? In-depth knowledge of AWS Glue, including job authoring, triggers, and cost-aware configuration. ? Familiarity with distributed data formats (Parquet, Avro), data lakes (Iceberg, Delta Lake), and cloud storage systems (S3, GCS, Azure Blob). ? Hands-on experience with Docker, Kubernetes, and CI/CD … to support and coach internal teams. Key Indicators of Success: ? Spark jobs are performant, fault-tolerant, and integrated into InfoSum's platform with minimal overhead. ? Cost of running data processing workloads is optimized across cloud environments. ? Engineering teams are equipped with best practices for writing, deploying, and monitoring Spark workloads. ? Operational issues are rapidly identified and resolved, with More ❯

via Docker/Kubernetes and integrate with orchestration systems (e.g., Airflow, custom schedulers). ? Work with platform engineers to embed Spark jobs into InfoSum's platform APIs and data pipelines. ? Troubleshoot job failures, memory and resource issues, and execution anomalies across various runtime environments. ? Optimize Spark job performance and advise on best practices to reduce cloud compute and … in at least two major cloud environments (AWS, GCP, Azure). ? In-depth knowledge of AWS Glue, including job authoring, triggers, and cost-aware configuration. ? Familiarity with distributed data formats (Parquet, Avro), data lakes (Iceberg, Delta Lake), and cloud storage systems (S3, GCS, Azure Blob). ? Hands-on experience with Docker, Kubernetes, and CI/CD … to support and coach internal teams. Key Indicators of Success: ? Spark jobs are performant, fault-tolerant, and integrated into InfoSum's platform with minimal overhead. ? Cost of running data processing workloads is optimized across cloud environments. ? Engineering teams are equipped with best practices for writing, deploying, and monitoring Spark workloads. ? Operational issues are rapidly identified and resolved, with More ❯

AWS Data Engineer – Contract Location: Reading (Hybrid – 1–2 days/month onsite) Rate: £500-550/day (Inside IR35) Start Date: ASAP Duration: 5 months (with potential for extension)A leading financial services organisation is seeking an experienced AWS Data Engineer to join their Compliance Reporting team. This backend-focused role involves designing and deploying … scalable data solutions that support the delivery of regulatory compliance reports across the business.You’ll work with a modern AWS stack and infrastructure-as-code tools to build robust data pipelines and applications that process complex datasets from multiple operational systems.Key Responsibilities: Build and maintain AWS-based ETL/ELT pipelines using S3, Glue (PySpark/… Python), Lambda, Athena, Redshift, and Step Functions Develop backend applications to automate and support compliance reporting Process and validate complex data formats including nested JSON, XML, and CSV Collaborate with stakeholders to deliver technical solutions aligned with regulatory requirements Manage CI/CD workflows using Bitbucket, Terraform, and Atlantis Support database management and improve data ingestion More ❯

AWS Data Engineer - Contract Location: Reading (Hybrid - 1-2 days/month onsite) Rate: £500-550/day (Inside IR35) Start Date: ASAP Duration: 5 months (with potential for extension) A leading financial services organisation is seeking an experienced AWS Data Engineer to join their Compliance Reporting team. This backend-focused role involves designing and deploying … scalable data solutions that support the delivery of regulatory compliance reports across the business. You'll work with a modern AWS stack and infrastructure-as-code tools to build robust data pipelines and applications that process complex datasets from multiple operational systems. Key Responsibilities: Build and maintain AWS-based ETL/ELT pipelines using S3, Glue … PySpark/Python), Lambda, Athena, Redshift, and Step Functions Develop backend applications to automate and support compliance reporting Process and validate complex data formats including nested JSON, XML, and CSV Collaborate with stakeholders to deliver technical solutions aligned with regulatory requirements Manage CI/CD workflows using Bitbucket, Terraform, and Atlantis Support database management and improve data More ❯

For more information, contact Mieke Van Tonder at . This is an exciting opportunity for someone passionate about technology, data, and machine learning, eager to contribute to a collaborative environment. The Data Scientist will build and deploy machine learning models to support personalization, recommendations, anomaly detection, and data insights. The role involves working with … as monitoring, continuous integration, and automated retraining. Utilize AI-assisted development tools like Cursor and Copilot to improve productivity. Collaborate with engineers, DevOps, and leadership to ensure robust data pipelines and translate business needs into technical solutions. Requirements At least 3 years of experience in applied machine learning and deploying production models. Proficiency in Python, SQL, and frameworks … learn. Experience with AWS services and Databricks; understanding of ML Ops is highly beneficial. Ability to adapt quickly to new tools and deliver scalable solutions independently. Familiarity with data pipelines involving Kafka, Debezium, S3, Lambda, and Delta Lake is a plus. Note: Some irrelevant content and personal opinions were removed for clarity and focus. This job posting is More ❯

in architecting our real-time mindshare platform that turns noisy social feeds into crystal-clear token insights. Your work will power a product so fast and intuitive, ingesting data from every channel, powering advanced NLP models, and delivering sub-second API responses that will deliver a trading experience that end users will only describe as magic. This role … deliver new ML-driven insights and to design intuitive consumer-facing dashboards and alert systems that visualize real-time mindshare and sentiment metrics aligned with product goals. Ensure data security and compliance, particularly around usergenerated content, API keys, and any PII in social media streams. Maintain code and model quality: author clean, efficient, and maintainable code; implement comprehensive … designing feedback loops, active learning, or user-in-the-loop systems to continuously improve model relevance. Experience with Git-based workflows and integrating model training into CI/CD pipeline Ideal Candidate Profile A creative problem-solver who is eager to innovate and push boundaries in the DeFi space. Deep expertise in data engineering and ML pipelines More ❯

BI Developer/Data Analyst (Power BI) Location: West Midlands Salary: Up to £65,000 Type: Full-time, Office-based About the Role An established food manufacturer and supplier is implementing their first-ever ERP system, focusing on four key modules: Manufacturing, Compliance, Reliability, and Learning. Each module will take approximately 4-6 months to complete. To support … and maintain Power BI dashboards and reports delivering actionable insights. Collaborate with internal stakeholders and external implementation partners throughout the ERP rollout and ongoing system maintenance. Develop robust data pipelines and write advanced SQL queries for ETL processes and data analysis. Migrate and integrate data between systems to ensure seamless communication and reporting. Continuously … monitor and improve BI tools post-ERP implementation. Analyse large datasets to uncover trends supporting strategic decision-making. Ensure data integrity, security, and automation of workflows to maintain reliability and efficiency. Benefits Competitive salary up to £65,000 25 days holiday plus bank holidays Onsite parking available Employee rewards and retail discounts Pension scheme (details to be confirmed More ❯

to the next level and be part of a global innovative team shaping the future of betting entertainment for millions worldwide. The Goal We're looking for a Data Engineer/AI Platform Specialist to build and optimize the infrastructure powering our AI ambitions. In this hands-on role, you'll design scalable data pipelines, manage … cloud services, and implement infrastructure-as-code to enable secure, efficient, and high-performance AI workflows. Your work will ensure data is not just available-but ready for AI at scale. You'll collaborate closely with ML engineers, software developers, and stakeholders to unlock siloed data, improve compliance, and drive platform reliability. This is your chance … the technical foundation behind real-world AI impact in a high-growth, high-regulation industry. The Player You bring deep expertise in cloud infrastructure (AWS, Azure, or GCP), data engineering, CI/CD, and IaC tools like Terraform. You thrive in cross-functional environments, balance performance with governance, and know how to turn complex systems into seamless pipelines. More ❯

risk and ensuring auditable adherence to complex financial laws in a high-volume, regulated environment Key Responsibilities: Design, build, and operate highly scalable, resilient, and secure microservices and data processing systems using idiomatic Golang. Model complex, long-running tax reporting business logic as durable, stateful workflows using Temporal to ensure correctness and reliability. Implement and optimize data pipelines consuming financial events from Redpanda, ensuring data quality, timeliness, and integrity. Collaborate with cross-functional teams of tax analysts, legal experts, and product managers to translate complex and evolving regulatory requirements into robust, maintainable, and verifiable software solutions. Provide technical leadership and mentorship, driving best practices in distributed system design, API definition with gRPC, code … quality, and advanced testing strategies. Build systems with auditability, traceability, and data lineage as a first-class concern to ensure transparency for our auditors and regulators. Own the full lifecycle of your services, including automated deployment (CI/CD with tools like Jenkins and Terraform), proactive monitoring, and incident response, following a "you build it, you run it More ❯

on highly impactful problems Promote a positive culture of collaboration, through open and effective communication, particularly when addressing issues or raising concerns. Are able to form well reasoned, data driven or otherwise evidence based arguments, to influence key stakeholders across the business. Required Skills and Experience Has 5+ years commercial experience Expert level in (Javascript Typescript Python) Familiar … with Postgres and K8s Feels at home in AWS console Has built infrastructure with Terraform Bonus points Has worked in an intelligence collection setting Experience with "big data" technologies, the management of data, and data pipelines Familiarity with functional programming concepts Has run production workloads of 1000s QPS Has been part of an "on More ❯

Deployed Engineers sit at the intersection of product, engineering, and customer success. You'll own full-stack features end-to-end, with a focus on building for enterprise data requirements. You will collaborate closely with customer teams to architect and implement sophisticated data pipelines and APIs, directly fueling our cutting-edge agentic AI with terabytes of … and growing your AI skills in a truly AI-first company at the forefront of agentic systems. Tasks What You'll Do Design & build scalable backend services and data pipelines - written in Python and deployed with Docker & Kubernetes. Integrate with enterprise ecosystems -enterprise software systems such as SAP and Oracle ERP, GraphQL/REST APIs, SFTP feeds, and … event buses (Kafka, Pulsar). Wrangle large, heterogeneous data sets -model, transform, and index multi-modal, multi-terabyte enterprise datasets for advanced workloads Develop enterprise-level next generation AI systems with the support of Magentic's AI specialists Ship complete customer features -from architecture and code to CI/CD, infra-as-code (Terraform), rollout, and user training. More ❯

Product About Cognitive Credit: Cognitive Credit is a technological pioneer transforming global credit markets through innovative data and analytics. We partner with the majority of the world's leading buy-side and sell-side institutions, providing unparalleled insights into investment grade bond, high yield bond, and leveraged loan markets, across the US and Europe. By delivering the fastest … most accurate credit data paired with specialist research software, we empower our clients to navigate an ever-more-digitalized global credit market with agility and confidence. Role Summary: We are seeking an experienced Quantitative Analyst to lead our quantitative research and modelling efforts. In this role you will drive innovation and growth across our product lines by applying … a quant finance framework to our proprietary, market-leading datasets. You will be collaborating with our Data and Engineering teams and working with clients to enhance our product offerings and grow our emerging quant services. The successful candidate will own the development of market valuation models and incorporate additional predictive analytics components across our web application, to support More ❯

to humanity's enduring challenges. We are looking for a Senior DevSecOps Engineer to join the Pathogen Programme at EIT. In this role, you'll help ensure our data platform is built to the highest standards, with a strong emphasis on automation across the development lifecycle. You'll work closely with engineers to deploy data pipelines … their workflows. You'll be responsible for maintaining infrastructure, designing secure automation pipelines, managing cloud environments, and ensuring security and compliance. You'll collaborate with cross-functional teams, data engineers, backend, and full-stack developers, to build robust, automated deployment pipelines across our environments. Key Responsibilities Design, implement, and maintain secure cloud infrastructure using Oracle Cloud Infrastructure (OCI … tools like Terraform to enable secure, repeatable deployments. Implement and manage CI/CD pipelines, focusing on automated security testing, deployment, and monitoring. Ensure all aspects of the data platform OCI infrastructure, data ingest pipelines, tool deployments, access controls, and monitoring are developed, tested, and deployed using automation best practices. Support bioinformaticians in building pipelines that More ❯

Requirements The engineering challenge is building systems reliable enough to power these high-stakes decisions. You'll work across data pipelines processing billions of tokens, real-time simulation platforms, and B2B SaaS products that make complex AI feel intuitive , (Desirable) Experience with Python, TypeScript/JavaScript, PostgreSQL, or Google Cloud Platform , Strong software engineering fundamentals with experience in … modern programming languages , (Desirable) Background in building scalable systems, APIs, and data-intensive applications , Proven track record of shipping production software and taking ownership of systems end-to-end , (Desirable) Familiarity with AI/ML systems, large language models, or high-throughput data processing , Experience working collaboratively in cross-functional teams with diverse technical backgrounds , (Desirable … monitoring, testing, and GitHub Actions CI/CD pipelines to maintain high availability for systems that influence million-pound business decisions , Collaborate Across Disciplines: Partner closely with our data science team to translate research innovations into robust, scalable production systems , Drive Technical Excellence: Contribute to engineering standards and systematic approaches as we scale our platform and grow the More ❯

working £300 to £350 a day A top timer global consultancy firm is looking for an experienced Hadoop Engineer to join their team and contribute to large big data projects. The position requires a professional with a strong background in developing and managing scalable data pipelines, specifically using the Hadoop ecosystem and related tools. The role … will focus on designing, building and maintaining scalable data pipelines using big data hadoop ecosystems and apache spark for large datasets. A key responsibility is to analyse infrastructure logs and operational data to derive insights, demonstrating a strong understanding of both data processing and the underlying systems. The successful … candidate should have the following key skills Experience with Open Data Platform Hands on experience with Python for Scripting Apache Spark Prior experience of building ETL pipelines Data Modelling 6 Months Contract - Remote Working - £300 to £350 a day Inside IR35 If you are an experienced Hadoop engineer looking for a new role then this is More ❯

About Route Reports Route Reports builds cutting-edge devices and software for inspecting roads, railways, and airports. We empower infrastructure teams with real-time data and analytics to keep infrastructure safe and costs down. Our AI hardware and software solution is market-leading in the UK, and we work with clients like Network Rail, Gatwick Airport, Essex Highways … develop the platform. Key Responsibilities Architect and implement new features across our React front end and Python/JS backend. Build and integrate with RESTful APIs, microservices, and data pipelines Optimise application performance, load times, and reliability Collaborate on UX design, sprint planning, and code reviews Ensure security best practices and compliance with data-protection standards More ❯

Are you a highly skilled Senior Backend Engineer with a passion for Data and Machine Learning, ready to make a global impact? Join Spotify's Commerce Platform to build the next generation of our robust, high-performing, and resilient payments ecosystem. This is a unique opportunity to solve complex challenges, work with ML-powered systems, and contribute to … prevention. The reliability of these systems is business-critical for Spotify. What You'll Do Architect, design, and implement highly scalable backend services (Java/Python) and robust data pipelines that power Spotify's internal Commerce platform. Develop and enhance our ML-powered systems taking solutions from concept to production. Take the lead on API design, platform development … and ensuring the scalability, reliability, and performance of our services. Collaborate with a talented, cross-functional team of engineers, product managers, and data scientists to deliver impactful solutions. Mentor other engineers and foster a culture of continuous learning and technical excellence within the team. Who You Are You have significant experience building and scaling backend services (Java and More ❯

models, we build the driving intelligence that feeds them. Our mission is to drive safer roads and fairer insurance through data. The Telematics team transforms raw phone sensor data into meaningful insights about how people drive. Using advanced signal processing and machine learning, we process high-frequency data to extract behavioural signals that power our understanding … of driving quality, context, and risk. About the Role As a Lead Data Scientist, you'll drive the technical direction of our behavioural modelling work. You'll lead the development of telematics features from idea to production, including exploratory analysis, signal design, risk evaluation, and scalable deployment. This is a hands-on leadership role where you'll design … things into production, values strong engineering, and has opinions on building reliable, scalable systems. What you will be doing Drive the development of telematics features from mobile sensor data, including project design, modelling, validation, and deployment. Conduct deep dives into driving signals, develop hypotheses, and evaluate their predictive power for risk. Design technical solutions with clear trade-offs More ❯

business units. ️ Design AI-based architectures and recommend appropriate tools, technologies, and methodologies ️ Collaborate with business leaders to understand objectives and translate them into scalable AI models ️ Architect data pipelines to support machine learning workflows and model deployments ️ Evaluate and implement deep learning, computer vision, NLP, and predictive modeling solutions ️ Conduct risk assessments on AI implementations and mitigate … compliance or ethical concerns ️ Partner with DevOps to integrate AI models into production systems with CI/CD ️ Define enterprise AI standards and ensure reusability of components ️ Mentor data scientists and engineers in applying best practices ️ Present technical documentation and demos to internal and external stakeholders ️ Build prototypes to validate AI use cases and accelerate adoption ️ Ensure AI More ❯

different types of databases: Relational, Graph etc Design and optimise APIs using Python and FastAPI to serve AI solutions. Familiar with GCP ecosystem and Cloudrun Build and optimise data pipelines for vector search and knowledge retrieval using Vector databases and embedding models. What We're Looking For: Professional AI engineering experience. Background in Software Engineering with Python. Solid … Excellent communication skills and the ability to work well in a collaborative team environment Nice to have: Strong experience with GCP Experience with Graph databases Experience in bringing data-intensive projects into production Experience with CI/CD pipelines Benefits: Private Health Insurance: Comprehensive coverage for both physical and mental health. Flexible and Remote-First Work Environment: Choose More ❯

OpenTofu, Terragrunt or Pulumi, we want to hear from you. This isn't just about building new things. You'll be enhancing and maturing one of thelargest health data platforms in the world, driving improvements, fixing bugs, and supporting a system that truly matters. You'll be hands-on with infrastructure-as-code, GitOps workflows, and engineering best … will include but not limited to the participant-facing websites, import and processing of high-volume health, NHS and genetic datasets to the de-identification/sharing of data into accredited Trusted Research Environments (TREs). At Our Future Health, our mission is to transform the prevention,detectionand treatment of conditions such as dementia, cancer, diabetes, heart disease … Hands-on experience working directly with software engineering best practices: unit testing, code reviews, design documentation, excellent debugging, troubleshooting skills. Experience in building/deploying tools related to data pipelines, ETL processes. Confident with cloud-native technologies like Kubernetes, Docker. Experience deploying open-source technologies such as Python, Node.JS, Ruby, Postgres and related CI/CD pipelines. Good More ❯

BI Developer/Data Analyst (Power BI) Location: West Midlands Salary: Up to £65,000 Type: Full-time, Office-based About the Role An established food manufacturer and supplier is implementing their first-ever ERP system, focusing on four key modules: Manufacturing, Compliance, Reliability, and Learning. Each module will take approximately 4-6 months to complete. To support … and maintain Power BI dashboards and reports delivering actionable insights. Collaborate with internal stakeholders and external implementation partners throughout the ERP rollout and ongoing system maintenance. Develop robust data pipelines and write advanced SQL queries for ETL processes and data analysis. Migrate and integrate data between systems to ensure seamless communication and reporting. Continuously … monitor and improve BI tools post-ERP implementation. Analyse large datasets to uncover trends supporting strategic decision-making. Ensure data integrity, security, and automation of workflows to maintain reliability and efficiency. Benefits Competitive salary up to £65,000 25 days holiday plus bank holidays Onsite parking available Employee rewards and retail discounts Pension scheme (details to be confirmed More ❯