1 to 25 of 85 Data Pipeline Jobs in Milton Keynes

Key Responsibilities: Data Pipeline Development: o Develop and maintain data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, files) into the Azure Databricks platform. o Implement data extraction, transformation, and loading (ETL) processes. o Ensure data pipelines are efficient, reliable, and scalable. Data Transformation & Processing: o Implement data transformations using Spark (PySpark or Scala) and other relevant tools. o Perform data cleaning, validation, and enrichment. o Ensure data quality and consistency. Azure Databricks Implementation: o Work … in team meetings and code reviews. Automation & DevOps: o Build CI/CD pipelines, for environmental deployments. o Contribute to automating data pipeline deployments and other data engineering tasks. o Learn and apply DevOps principles. More ❯

Job Title: Senior Data Engineer Key Requirements: 4-8 years of experience from tier 1 or 2 big tech companies Job Location: Abu Dhabi, UAE Benefits: Work with cutting-edge technology through modern infrastructure and automation projects Thrive in a growth-focused environment that prioritizes learning, innovation, and … role requires a deep understanding of modern data engineering practices, real-time processing, and cloud-native solutions. Key Responsibilities: Data Pipeline Development & Management: Design, implement, and maintain scalable and reliable data pipelines to ingest, transform, and load structured, unstructured, and real-time data … environments. Machine Learning & Advanced Analytics Enablement: Collaborate with data scientists to prepare and serve features for ML models. Maintain awareness of ML pipeline integration and ensure data readiness for experimentation and deployment. Documentation & Continuous Improvement: Maintain thorough documentation including technical specifications , data flow More ❯

Data Management Specialist (Azure) - Newcastle/Hybrid You will provide expert-level troubleshooting, root cause analysis, and performance optimisation for data platforms,, such as Azure SQL databases, Fabric Warehouse and Onelake, Databricks, and Azure Data Factory. Purview or another MDM tool is highly desirable. … Skills/experience Bachelor’s degree in Information Technology, Computer Science, Data Management, or a related field. 4+ years of experience in data management, data operations, or IT support roles, with at least 2 years in Level 3/4 support or equivalent. Hands … on experience with database technologies ( Azure landscape , Oracle, PostgreSQL, etc.). Desirable: Industry certifications in professional data management, cloud platforms (AWS, Azure, GCP), metadata management, or database administration (e.g., Microsoft Certified: Azure Data Engineer Associate, DAMA CDMP) are desirable. Proven experience with ETL tools, data More ❯

Role: Data Engineering Analyst(Python & Power BI) Location: Leeds/Edinburgh(Hybrid) Duration: Contract About Us: We're seeking a skilled Data Engineering Analyst to join our team. You'll play a key role in designing, building, and maintaining our data infrastructure, ensuring seamless … data flow and insightful analytics. Key Responsibilities: Design, develop, and maintain data pipelines using Google Cloud Platform (GCP) services like BigQuery, Cloud Storage, and Cloud Dataflow. Develop and maintain reports, dashboards, and visualizations using Power BI, ensuring data-driven insights for business stakeholders. Write … and reporting. Strong programming skills in Python, with experience in data processing libraries like Pandas and NumPy. Experience with data pipeline development, data warehousing, and ETL processes. Strong analytical and problem-solving skills, with attention to detail and ability to work in a More ❯

Key aspects of the role include: Data Pipeline Development: Architect, develop, and optimise data extraction, transformation, and loading (ETL) processes for high-volume healthcare data. Integration of NHS Systems Data: Work directly with NHS data sources and ensure seamless integration of … systems like UDAL, NCDR, SUS+, HES, and ODS into our analytics platform. System Performance: Monitor and enhance the performance of our data systems to ensure reliability and responsiveness. Collaboration: Partner with data scientists, analysts, and stakeholders to ensure that data solutions align with business … needs and analytic strategies. Scalability & Security: Implement and enforce best practices for data security, quality, and scalability across our data infrastructure. Essential Skills: Healthcare Data Expertise: Proven experience in designing and managing data solutions within healthcare environments. Demonstrated ability to work with More ❯

the ease of use - they have introduced several industry-first products and maintain a strong focus on research and development. As a Junior Data Engineer, you will play a crucial role in developing the data infrastructure that supports the analytics and operational functions of our business … enterprise. Role & Responsibilities: Data Infrastructure & Support: Provide first-line support for data engineering tasks, such as managing and monitoring data pipelines, resolving issues, and ensuring data integrity. Work with both structured and unstructured datasets to design and implement data models … -related errors. Collaborate with business stakeholders to identify data requirements and deliver sustainable solutions. Monitor and troubleshoot data pipeline issues to maintain data integrity and accuracy. Assist in the development, maintenance, and optimization of ETL (Extract, Transform, Load) processes for efficiency More ❯

join us at FPSM and be part of something bigger... What we are looking for: We are seeking a highly motivated and experienced Data Product Manager to own the strategy, roadmap, and execution of our Customer Data Platform (CDP) and Single Customer View (SCV). This … is a critical role within our rapidly growing data organization, and you will be instrumental in enabling data-driven decision-making across the business. You'll work closely with engineering, marketing, product, and other internal teams to ensure our CDP is a powerful tool for understanding … with overall business objectives and stakeholder needs. Requirements Gathering: Proactively gather, analyze, and prioritize requirements from internal stakeholders (marketing, product, analytics, etc.) for data integration, access, and utilization. Data Source Integration: Oversee the integration of various data sources into the CDP, ensuring data More ❯

Azure Data Engineer - Migrations Largely remote - 1 day per month in the office Salary up to £55,000 + Bonus + Benefits We are looking for a hands-on Data Engineer/Migration Engineer to support the migration of CRM data from various OpCo … of legacy source systems, into the Microsoft Dynamics 365 Customer Engagement (CE) platform. You will be responsible for extracting, transforming, cleansing, and loading data , ensuring quality and consistency throughout the process. This role is ideal for someone with a strong technical background in data engineering and … experience with Microsoft/Azure data tools, particularly in a CRM or Dynamics/D365 environment. In the medium term, there will be scope and opportunity to build modern, cloud-native data solutions using Azure Synapse or Microsoft Fabric. Key Responsibilities As the sole Data More ❯

Azure Data Engineer - Migrations Largely remote - 1 day per month in the office Salary up to £55,000 + Bonus + Benefits We are looking for a hands-on Data Engineer/Migration Engineer to support the migration of CRM data from various OpCo … of legacy source systems, into the Microsoft Dynamics 365 Customer Engagement (CE) platform. You will be responsible for extracting, transforming, cleansing, and loading data , ensuring quality and consistency throughout the process. This role is ideal for someone with a strong technical background in data engineering and … experience with Microsoft/Azure data tools, particularly in a CRM or Dynamics/D365 environment. In the medium term, there will be scope and opportunity to build modern, cloud-native data solutions using Azure Synapse or Microsoft Fabric. Key Responsibilities As the sole Data More ❯

ideally, or any part of the world, as long as they are willing to relocate to KL, Malaysia. Are you passionate about using data to drive innovative solutions in a fast-paced environment? We're looking for a Senior Data Engineer to join a cutting-edge … technology company based in Kuala Lumpur! As a Senior Data Engineer, your mission will be to support data scientists, analysts, and software engineers by providing maintainable infrastructure and tooling for end-to-end solutions. You’ll work with terabytes to petabyte-scale data, supporting … multiple products and data stakeholders across global offices. Key Responsibilities Design, implement, operate, and improve the analytics platform Design data solutions using various big data technologies and low latency architectures Collaborate with data scientists, business analysts, product managers, software engineers and other More ❯

Azure Data Engineer/Consultant – Up to £75K + Bonus Location: Remote (UK-based, with very occasional travel) Salary: Up to £75,000 + bonus Company & role Join one of the UK’s most advanced Microsoft partners, specialising in Azure-led Data & AI solutions. This consultancy … is known for transforming data into tangible value through innovative platforms and strategic data services. With a collaborative team of experts, they deliver impactful solutions across enterprise clients, leveraging the very latest in Microsoft technology. Why This Role Stands Out Play a key role in shaping … across the Azure Data Platform (Fabric, Synapse, Databricks, Data Lake, Data Factory, Purview, etc.) Strong data pipeline and ETL skills, including data modelling and transformation Experience implementing Medallion Architecture (Bronze, Silver, Gold layers) in lakehouse or Databricks environments Familiarity More ❯

Azure Data Engineer Remote (UK-based) Up to £60,000 per annum About the Company We are a leader in the insurance sector, committed to data-driven decision making and modernising our technology landscape. As part of our continued growth, we’re seeking a skilled Azure … Data Engineer to support our data platform and engineering initiatives. Role Overview As an Azure Data Engineer, you will be instrumental in designing, building, and optimising data pipelines and architecture in our Azure environment. This role is ideal for someone with strong … SQL skills and a background in Azure data services, looking to play a key role in transforming and scaling our data operations. Key Responsibilities Develop and maintain scalable data pipelines in Azure. Build and optimise data integration workflows using Azure Data More ❯

in the UK.Calex is the supplier of learning services, working in partnership with Volkswagen Group UK. The role: We are looking for a Data & Reporting Team Manager who is an expert in their field of Data Architecture/Data Science, with a proven record … of delivering innovative solutions in a fast paced environment, a meticulous level of attention to detail and passion for creating robust, user centric data and reporting environments. This role will report directly to the Learning Operations Manager and manage a team of Data Engineers/Analysts. … Basic Json SQL Database Schema and security Experience with database schemas and security Extensive experience of working with: Azure Data Factory – Pipelines, Data Flows and Sink Azure Generation 2 Data Lake and Data Blobs Power Query Excel Experience of working with Power More ❯

Data Engineer/Data Architect Location: Penryn, Cornwall- On premise About Us: Aspia Space is building the next generation of planetary intelligence. We transform observational data into trusted, consumable intelligence for agriculture, finance, environmental planning and policy. From our offices in Harwell and Cornwall … we push Earth data to its limits to deliver insights no one else can, developing products that combine satellite imagery and expert ground truth with the very best observational science and deep learning tools to support clients across the world, powering everything from smallholder farming programmes in Africa … impact. Our ambition is global, but our focus is always local, measurable, and relevant. Role Overview: We’re looking for a highly skilled Data Engineer/Data Architect who can hit the ground running and join us in our Penryn office. You will be instrumental in More ❯

Position Overview Our client, a leading player in the insurance sector, is seeking a seasoned Azure Data Factory (ADF) Architect to spearhead the design, implementation, and optimization of data pipelines within the Azure ecosystem. This pivotal role involves providing strategic consulting and technical leadership to ensure … ADF features, troubleshooting, and continuous improvement of data workflows in production environments. Mentor Teams: Lead and mentor development teams on secure ADF pipeline development, API management, and compliance with enterprise security standards. Ensure Best Practices: Establish and enforce best practices for data integration, transformation, and … governance within the Azure platform. Required Qualifications Technical Expertise: Extensive experience with Azure Data Factory, including pipeline design, data flow, and integration runtime configuration. Proficiency in Azure services such as Azure Data Lake, Azure Synapse Analytics, Azure SQL Database, and Azure Key Vault. More ❯

Data Engineer - Leading Pharmaceutical Company - Manchester (Tech Stack: Data Engineer, Databricks, Python, Power BI, AWS QuickSight, AWS, TSQL, ETL, Agile Methodologies) About the Role: We are seeking a talented and experienced Data Engineer on behalf of our client, a leading Software House. This is … work with cutting-edge technologies and contribute to exciting projects in a collaborative environment. About the Role: Our client is seeking an experienced Data Engineer to join their team in Manchester. This hybrid position involves working within the pharmaceutical industry, focusing on the design, development, and maintenance of … data pipelines, ETL processes, and databases. The role is ideal for someone passionate about improving processes, ensuring data quality, and maintaining compliance with regulatory standards. focusing on designing, developing, and maintaining data pipelines, ETL processes, and databases. If you are passionate about driving continuous More ❯

A high performing, multi-billion dollar AUM hedge fund are looking to make a newly created Data Analyst hire into their rapidly growing centralised Data Management team. Reporting into the Global Head of Data and taking on a highly visible role within the team … the hire will be working very closely with Operations, Product Control, Risk, Trading and Research teams to ensure the smooth transmission of data, in … a role that has a highly technical/engineer angle. You will optimize the operation and quality management of the internal data pipeline with oversight of raw data initially being onboarded through to the delivery of datapoints to front office/investment teams. You will More ❯

advertised date, so please apply as soon as you can. What you’ll be doing Contribute to the design and execution of the data migration and integration strategy within the Mortgages area, supporting the broader integration programme objectives. Collaborate with internal stakeholders, business analysts, and architects to translate … functional and non-functional requirements into technical data solutions. Design, build and maintain robust, scalable and high-performance ETL/ELT pipelines using modern data engineering tools and practices, ensuring quality, lineage and auditability. Support data discovery activities and analysis to inform data mapping, profiling, and migration planning, especially in the context of mergers or acquisitions. Develop and maintain detailed technical design documents and data flow diagrams aligned with architectural principles and data governance standards. Assist in the implementation and enforcement of best practices in data More ❯

collaboration, recognition and inclusivity and offer a wide range of benefits to support staff wellbeing. Your Future Starts Here PURPOSE OF JOB: The Data & AI Solutions Deployment Architect will be instrumental in architecting and deploying Azure Data & AI Solutions. A solid, technical, ‘hands-on’ (not just … grasp of Azure Fabric is essential as we navigate clients through all of the infrastructure, coding, and product development lifecycle challenges around successful Data and AI implementation and deployment. This customer-centric position involves delivering projects, conducting workshops and physically working on client’s environments via Python, and … C#. This role would suit an experienced Data platform engineer or Data Solutions Architect who thrives in working across multiple projects with a diverse range of clients, and who places the importance of bridging the gap between complex data problems and real life product More ❯

Job Title: Senior Data Engineer Salary: up to £65,000 Location: Hybrid – 1 day per fortnight in the Wilmslow offices Working hours 0900 to 1730 The Role Following recent investment, The Citation Group are looking to build out their data capabilities to be Best in Class. … As such, a fantastic opportunity has arisen for a strong data engineer to work in the data engineering team, to expand and maintain the data infrastructure that provides the foundations from which our BI, Data Science and Operational Data teams … insights and revenue to the functions across the wider group. We’re looking for an individual who has experience working with enterprise grade data pipelines and infrastructure, who can implement processes on our modern tech stack with robust, pragmatic solutions. Responsibilities develop, and maintain ETL/ELT data More ❯

Role Title: Lead Data & Insights Engineer Salary: £65,00-£80,000 Location: UK Remote Full-time | Permanent | 37.5 hours/week Apply by: 12/06/2025 Hiring Manager : Phil Walter | Recruiter: Rikki Spencer-Cox Help us “Elevate Education, Everywhere.” Team: Technology & Digital | Etio At Etio, we … We manage both internal and partner delivery streams, bringing resilience to complex systems and driving continuous improvement across live services. The Role: Lead Data & Insights Engineer We're looking for a strategic and delivery-focused data leader to join Etio as our Lead Data … who combines deep technical expertise with strong autonomy and a sharp eye for operational value. This is the perfect opportunity for a senior data professional who wants to own and evolve data architecture across a growing organisation. You’ll act as the primary in-house expert More ❯

Senior Data Engineer Bristol 6-month contract Paying up to £80p/h (Outside IR35) *** Please note that due to the nature of the work you must hold or be eligible to obtain DV level security clearance *** Responsibilities : Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. Implement data ingestion, transformation, and integration processes, ensuring data quality and security. Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards. … Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity. Develop robust data models to support analytics and reporting within secure environments. Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack. More ❯

Job Title: Snowflake Data Architect Job Location: Hove, UK/Hybrid Is it Permanent/Contract: Permanent Job Description Who we are? We are a leading global IT Services company, dedicated to driving digital transformation and innovation for businesses around the world. Founded in 1990, Client has grown … and excellence. We foster a collaborative environment where creativity and continuous learning are encouraged, enabling our employees to thrive and grow. The Opportunity. Data & AI practice is experiencing rapid growth, and we're on the lookout for delivery leaders. Our ideal candidates should be passionate about data … responsibilities in challenging engagements, gaining exposure to clients in a variety of industries across the UK and Europe. If you're a dynamic Data Solution Architect leader who excels in a collaborative team environment and enjoys working with talented individuals, this could be the perfect opportunity for you. More ❯

Data Engineer £50,000-£68,000 per annum Hybrid role with 3 days per week in Burton-on-Trent office Seeking a skilled Data Engineer to deliver the data engineering roadmap and help drive data maturity across the organisation. You'll be … responsible for the end-to-end data engineering lifecycle from ingestion and transformation to delivery while ensuring data solutions are scalable, secure and aligned with business goals. Key Responsibilities: · Design, build and maintain robust data pipelines and transformation logic · Own data engineering … batch, streaming, and APIs · Collaborate with cross-functional teams to understand and deliver on business needs · Ensure regulatory compliance, data quality, and pipeline documentation · Optimise the central data platform in partnership with architecture and analytics teams · Champion best practices in data engineering, MLOps More ❯

Data Engineer We’re working with a forward-thinking Microsoft partnered consultancy who are looking for a Data Engineer to join their growing team. You will have the opportunity to work with some of the latest Microsoft technologies with a focus on projects on Databricks implementations. … You will join a team at the centre of a number of data-driven projects where you will be responsible for the design, development and creation of data solutions. You will work on the full end-to-end product lifecycle from platform design to insights creation. … This is a fantastic opportunity for someone who is passionate about building scalable, high-performance data solutions utilising the latest cutting-edge cloud technologies. As part of this role, you will be responsible for some of the following areas • Design and develop scalable data pipelines to More ❯