1 to 25 of 44 Data Pipeline Jobs in the Midlands

Join our client in embarking on an ambitious data transformation journey using Databricks, guided by best practice data governance and architectural principles. To support this, we are recruiting for talented data engineers. As a major UK energy provider, our client is committed to 100% renewable energy and sustainability, focusing on delivering exceptional customer experiences. … day a week being based in their Nottingham office, this is negotiable. It is a full-time role, 37 hours per week. Accountabilities: * Develop and maintain scalable, efficient data pipelines within Databricks, continuously evolving them as requirements and technologies change. * Build and manage an enterprise data model within Databricks. * Integrate new data sources into … such as data warehousing. * Able to write repeatable, production-quality code for data pipelines, utilizing templating and parameterization where needed. * Can make data pipeline design recommendations based on business requirements. * Experience with data migration is a plus. * Open to new ways of working and new technologies. * Self-motivated with the ability More ❯

Job Title: Senior Data Engineer Location: Warwick, Hybrid - 2 days per week onsite Remuneration: Up to £650 per day Contract Details: Fixed Term Contract (6 months) Working Pattern: Full Time Join a leading organisation dedicated to connecting people to the energy they use safely, reliably, and efficiently! We're on the lookout for a talented Senior Data Engineer to elevate our data capabilities and drive our projects forward. If you're passionate about data engineering and thrive in a dynamic environment, we want to hear from you! Responsibilities: Lead the design, development, and maintenance of large-scale data systems, including data warehouses, data lakes, and … data pipelines. Create and manage ETL processes using tools like Matillion for seamless data integration. Develop and maintain data warehouse objects such as tables, views, and security roles in Snowflake. Extract data from various sources, ensuring accuracy and consistency throughout the process. Transform and cleanse data, implementing data More ❯

About Us Makutu designs, builds and supports Microsoft Azure cloud data platforms. We are a Microsoft Solutions Partner (Azure Data & AI) and are busy building a talented team with relevant skills to deliver industry leading data platforms for our customers. The Role The Data Engineer role is key to building and growing … technical team at Makutu. The role will provide the successful applicants with the opportunity for significant career development while working with a range of large businesses to whom data is critical to their success. Working as part of the team and with the customer, you'll require excellent written and verbal English language and communication skills. Big growth … plans are in place to build a broader and deeper technical capability with a focus on the Microsoft Azure technology stack. The position of Data Engineer is a key role in the wider capability of our team. Occasional visits to our Head Office and customers sites will be required. Key responsibilities: Identify, design, and implement working practices across More ❯

Data Engineer Permanent role Birmingham Based – Hybrid working Competitive Salary Based on Experience + Bonus scheme Amtis is proud to be recruiting for a Data Engineer to join a forward-thinking business building modern, scalable data solutions. This is a hands-on role focused on building robust data infrastructure that supports high … performance analytics. Responsibilities: Build scalable, reusable data pipelines using modern cloud technologies Design secure, flexible data models for internal and external users Optimise performance across large, complex datasets Collaborate with analytics and visualisation teams to align data flows Integrate external data sources using bespoke and automated pipelines Support CI/CD and … Factory, Blob Storage Python/PySpark SQL Server, Parquet, Delta Lake Deep understanding of: ETL/ELT, CDC, stream processing Lakehouse architecture and data warehousing Scalable pipeline design and database optimisation A proactive mindset, strong problem-solving skills, and clear communication If you're an experienced data engineer and interested in the opportunity, please More ❯

Senior Data/Power BI Engineer West Midlands/HybridPermanent - Full Time Are you passionate about turning data into powerful insights? TXP is looking for a Senior Data/Power BI Engineer to help our clients unlock the full potential of their data. In … this role, you'll: Design and deploy Power BI models and dashboards Collaborate with stakeholders to shape intelligent analytics solutions Support data warehousing and data pipeline development Drive analytics maturity and best practices across client organisations Mentor junior team members and lead by example What you'll bring: Expertise in Power BI, DAX, and SQL … Experience with Microsoft Fabric or similar platforms Strong understanding of manufacturing processes and ERP systems A passion for data storytelling and business impact Join a consultancy that values innovation, diversity, and continuous learning. Be part of a team that's shaping the future of data-driven decision-making. Apply now and help us build smarter, data More ❯

Lead Data Architect £67,000 - 76,000 GBP Hybrid WORKING Location: Birmingham, West Midlands - United Kingdom Type: Permanent Lead Data Architect Locations: Leeds, Manchester, Birmingham, Bristol, Glasgow, or London (Hybrid - up to 2 days on-site per week, remainder remote) Salary: £67,000 - £76,000 (non-London)/Up to £83,000 (London) + Benefits A … respected digital consultancy is hiring a Lead Data Architect to join its growing Data Engineering practice. Known for delivering modern, human-centred services across public and private sectors, this organisation supports clients through meaningful transformation by combining data, design, and engineering. This role is ideal for experienced architects who enjoy shaping cloud-native data platforms, working in multidisciplinary teams, and embedding data governance and architectural rigour across complex service ecosystems. The Role: As a Lead Data Architect, you'll play a key role in designing scalable, secure, and accessible data platforms that enable insight-driven decision-making. You'll work closely with engineering teams and client More ❯

Data Engineer//1 Day/Week Onsite Worcestershire//Initial 3 Months//Up to £400/day Outside IR35 REED Technology are working with a client who are seeking a contract Data Engineer to join their dynamic Data Engineering team. You'll play a key role in designing and … implementing complex data flows that power analytics and business intelligence (BI) systems across their growing financial services organisation. Key responsibilities Designing, building, and maintaining robust data pipelines and platforms Collaborating with BI, tech, and business teams to deliver scalable solutions Leading on the implementation of new data infrastructure Analysing data for integrity … and correlation across systems Translating stakeholder requirements into technical deliverables Producing and maintaining high-quality technical documentation Championing data engineering best practices and standards across the business Technical skills Cloud data platforms - Azure, AWS, or GCP (Azure preferred) Snowflake - Deep knowledge and hands-on experience Matillion - Expertise in ETL orchestration Data warehousing and advanced More ❯

Lead QA Engineer Backend Data ETL Agile Automation Our insurance client in Leicester are looking for a Lead Quality Engineer focused on testing backend systems/data migration/ETL/data quality. If interested, please see the details below: Overview of role: As Lead Quality Engineer, you ll lead testing efforts within an … Agile team, ensuring the quality of data pipelines and ETL processes. You ll define test strategies, mentor a high-performing team, and collaborate with stakeholders to align testing with business goals. Responsibilities include hands-on data validation, test automation, performance testing, and continuous improvement of frameworks. Salary: £65k + bonus + very strong benefits package Location … continuous career progression opportunities. Inclusive and innovative culture/Continuous improvement environment Recognised as one of the very best places to work in 2025 Experience Required: Experience in data warehouse testing, including ETL processes, data pipelines and data integration. Strong data quality testing expertise Proven ability to define and implement testing strategies More ❯

Lead QA Engineer | Backend | Data | ETL | Agile | Automation Our insurance client in Leicester are looking for a Lead Quality Engineer focused on testing backend systems/data migration/ETL/data quality. If interested, please see the details below: Overview of role: As Lead Quality Engineer, you’ll lead testing efforts within an … Agile team, ensuring the quality of data pipelines and ETL processes. You’ll define test strategies, mentor a high-performing team, and collaborate with stakeholders to align testing with business goals. Responsibilities include hands-on data validation, test automation, performance testing, and continuous improvement of frameworks. Salary: £65k + bonus + very strong benefits package Location … continuous career progression opportunities. Inclusive and innovative culture/Continuous improvement environment Recognised as one of the very best places to work in 2025 Experience Required: Experience in data warehouse testing, including ETL processes, data pipelines and data integration. Strong data quality testing expertise Proven ability to define and implement testing strategies More ❯

Our mission is to unleash the power of first-party data to craft personalised campaigns and brand experiences that customers love, driving exceptional results for our clients. As a fast-paced and innovative agency, we challenge ourselves and our clients to think big. If you're ready to make an impact and grow with a dynamic team, we … want to hear from you. We're looking for a strategic and technically skilled Tracking Expert & Data Analyst to build and lead our data and analytics capabilities. This pivotal role will focus on implementing advanced tracking solutions, creating insightful reports, and providing data-driven recommendations to optimise performance. If you're passionate about leveraging … data to drive results and delivering impactful insights, this role offers an exciting opportunity to shape the way we collect, analyse, and use data across a diverse range of clients. What You'll Do What You'll Do Leadership Lead cross-functional projects to uncover insights that drive strategic decisions and optimisation opportunities. Collaborate with marketing More ❯

Technical Data Engineer Location: Northampton or London/Hybrid 2-3 days on-site 6 Months + We are seeking a highly skilled and communicative Technical Data Engineer to join our team. The ideal candidate will have hands-on experience with modern data platforms and tools including Databricks, DBT, and Snowflake. You will play … a key role in designing, developing, and optimizing data pipelines and analytics solutions that drive business insights and decision-making. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks and DBT. Develop and optimize data models and transformations in Snowflake. Collaborate with cross-functional teams to understand data requirements and … deliver robust solutions. Ensure data quality, integrity, and governance across platforms. Troubleshoot and resolve data-related issues in a timely manner. Document processes, workflows, and technical specifications clearly and effectively. Required Skills & Experience: Proven hands-on experience with: Databricks (Spark, Delta Lake, notebooks) DBT (data modeling, transformations, testing) Snowflake (SQL, performance tuning, data More ❯

you're inspired to think big and bring your ambition to work every day, which is why, at British Airways the sky is never the limit. The role: Data Scientist/Senior Data Scientist Join British Airways as a Data Scientist in our cutting-edge Operations Decision Support (ODS) product team. This full-stack … role sits at the heart of our transformation, combining data science, machine learning, and optimisation to build industrialised decision-support software. If you're passionate about solving real-world problems through intelligent, data-driven solutions and thrive in a collaborative, agile environment-this is your opportunity to make a real impact. What you'll do: Design … with software engineering teams, product & change teams to turn their models into industrialized products that are used by the airline every day to make better decisions. Build robust data pipelines, automate data cleaning, and deliver scalable, production-grade Python code using best practices such as typing, testing, and modular design. Integrate models into orchestration frameworks (e.g. More ❯

Contribute to security engineering projects, transitions, and transformations. Work closely with security operations and associated security incident response systems Stay informed about emerging threats and security best practices. Data Ingestion and Enrichment: Configure Elasticsearch pipelines for data ingestion from various sources, primarily from Kafka Enhance data enrichment by integrating threat intelligence feeds and contextual … towards achieving these. Skills Required for the Role Security and Compliance with Elastic Security: Set up access controls, authentication, and encryption using Elastic Security features. Ensure compliance with data protection regulations. Detection Rule Development: Ability to create, test, and optimise detection rules to identify suspicious activities and potential threats based on the MITRE ATT&CK Framework Performance Tuning … with Elasticsearch and Logstash: Fine-tune query performance using Elasticsearch indices and mappings. Monitor Logstash pipelines and optimize resource utilization. Kibana Visualization and Monitoring: Leverage Kibana for data visualization, dashboards, and real-time monitoring. Create custom visualizations to track data quality metrics and system performance. Elastic integration Integration of the SIEM with critical systems to provide More ❯

Contribute to security engineering projects, transitions, and transformations. Work closely with security operations and associated security incident response systems Stay informed about emerging threats and security best practices. Data Ingestion and Enrichment: Configure Elasticsearch pipelines for data ingestion from various sources, primarily from Kafka Enhance data enrichment by integrating threat intelligence feeds and contextual … achieving these. Skills Required for the Role Essential: Security and Compliance with Elastic Security: Set up access controls, authentication, and encryption using Elastic Security features. Ensure compliance with data protection regulations. Detection Rule Development: Ability to create, test, and optimise detection rules to identify suspicious activities and potential threats based on the MITRE ATT&CK Framework Performance Tuning … with Elasticsearch and Logstash: Fine-tune query performance using Elasticsearch indices and mappings. Monitor Logstash pipelines and optimize resource utilization. Kibana Visualization and Monitoring: Leverage Kibana for data visualization, dashboards, and real-time monitoring. Create custom visualizations to track data quality metrics and system performance. Elastic integration Integration of the SIEM with critical systems to provide More ❯

Data Engineer (Minerva MMTAR Functional Core Build) Max Pay: £537 SC is required, Duration: 6 months initially Location: Telford: If local to Telford, requirement will be 2 days per week on site. If not local, expectation will be to attend once/twice a month. Via Umbrella Key Responsibilities: Design, development, and deployment of data integration … and transformation solutions using Pentaho, Denodo, Talend, and SAS. Architect and implement scalable data pipelines and services that support business intelligence and analytics platforms. Collaborate with cross-functional teams to gather requirements, define technical specifications, and deliver robust data solutions. Champion Agile and Scrum methodologies, ensuring timely delivery of sprints and continuous improvement. Drive DevOps practices … for CI/CD, automated testing, and deployment of data services. Mentor and guide junior engineers, fostering a culture of technical excellence and innovation. Ensure data quality, governance, and security standards are upheld across all solutions. Troubleshoot and resolve complex data issues and performance bottlenecks. Key Skills: SAS 9.4 (DI), SAS Viya 3.x (SAS More ❯

Data Warehouse Engineer Location: Newark Salary: £35,000-£40,000 A well-established organisation in the Newark area is seeking a Data Warehouse Engineer to take ownership of its centralised data platform. With a current team member transitioning out at the end of July, this role is ideal for someone confident in SQL, data warehousing, and Power BI, who can hit the ground running in a collaborative, technically strong environment. You'll join a close-knit data team and work alongside a colleague focused on reporting to deliver meaningful insights and maintain a robust, scalable data infrastructure. This position offers exposure to a wide range of ERP systems … and an opportunity to shape the business's approach to data and reporting. Key Responsibilities: Manage and maintain the central data warehouse across multiple ERP platforms Develop and optimise SQL queries to support reporting and data access Build and maintain Power BI dashboards and visualisations Collaborate with internal teams to understand reporting needs and More ❯

Viya 4. Support the migration of legacy SAS platforms to cloud-native SAS Viya 4 environments, including namespace separation and Kubernetes orchestration. Implement and manage configuration objects, service data, and translation objects within SAS environments. Collaborate with data engineers and DevOps teams to integrate SAS with Oracle, GitLab CI/CD pipelines, and other enterprise platforms. … Monitor and optimise platform performance, ensuring high availability and compliance with security standards. Provide technical support and troubleshooting for SAS users and developers. Automate environment provisioning and data transfers using approved EA integration patterns. Participate in community knowledge sharing and contribute to best practice documentation. Essential Skills and Experience: SAS Enterprise Guide (EG) – workflow orchestration and user support. … Viya 4 – platform configuration, migration, and containerisation (Kubernetes). SAS Visual Analytics (VA) and SAS Visual Investigator (VI) – dashboard and investigation tool support. Oracle – integration and data pipeline support. Configuration Management – experience managing service objects, deployment artefacts, and environment baselines. Testing Lifecycle Knowledge – including test strategy design, environment comparison utilities, and automated testing integration. Desirable Attributes: Experience More ❯

Viya 4. Support the migration of legacy SAS platforms to cloud-native SAS Viya 4 environments, including namespace separation and Kubernetes orchestration. Implement and manage configuration objects, service data, and translation objects within SAS environments. Collaborate with data engineers and DevOps teams to integrate SAS with Oracle, GitLab CI/CD pipelines, and other enterprise platforms. … Monitor and optimise platform performance, ensuring high availability and compliance with security standards. Provide technical support and troubleshooting for SAS users and developers. Automate environment provisioning and data transfers using approved EA integration patterns. Participate in community knowledge sharing and contribute to best practice documentation. Essential Skills and Experience: SAS Enterprise Guide (EG) – workflow orchestration and user support. … Viya 4 – platform configuration, migration, and containerisation (Kubernetes). SAS Visual Analytics (VA) and SAS Visual Investigator (VI) – dashboard and investigation tool support. Oracle – integration and data pipeline support. Configuration Management – experience managing service objects, deployment artefacts, and environment baselines. Testing Lifecycle Knowledge – including test strategy design, environment comparison utilities, and automated testing integration. Desirable Attributes: Experience More ❯

Viya 4. Support the migration of legacy SAS platforms to cloud-native SAS Viya 4 environments, including namespace separation and Kubernetes orchestration. Implement and manage configuration objects, service data, and translation objects within SAS environments. Collaborate with data engineers and DevOps teams to integrate SAS with Oracle, GitLab CI/CD pipelines, and other enterprise platforms. … Monitor and optimise platform performance, ensuring high availability and compliance with security standards. Provide technical support and troubleshooting for SAS users and developers. Automate environment provisioning and data transfers using approved EA integration patterns. Participate in community knowledge sharing and contribute to best practice documentation. Essential Skills and Experience: SAS Enterprise Guide (EG) - workflow orchestration and user support. … Viya 4 - platform configuration, migration, and containerisation (Kubernetes). SAS Visual Analytics (VA) and SAS Visual Investigator (VI) - dashboard and investigation tool support. Oracle - integration and data pipeline support. Configuration Management - experience managing service objects, deployment artefacts, and environment baselines. Testing Lifecycle Knowledge - including test strategy design, environment comparison utilities, and automated testing integration. Desirable Attributes: Experience More ❯

Excel VBA. Experience with CI/CD pipelines and Azure DevOps is also valued. Key Responsibilities: - Develop serverless applications using Cloud Functions (Python or Node.js) - Build and optimise data pipelines with Cloud Dataflow - Manage relational databases via Cloud SQL (PostgreSQL/MySQL) - Implement workflows using GCS triggers and IAM permissions - Integrate APIs using Apigee - Create Excel-based automation More ❯

ranks 189 on the Fortune 500. Further information about Lear is available at or on Position overview: As an Application Solution Architect, you will design and implement scalable, data-driven solutions using Palantir Foundry. You'll work closely with business stakeholders, data engineers, and developers to translate business needs into robust technical architectures that leverage Foundry … technical constraints. Design scalable and efficient data models and ontologies within Foundry. Define and oversee the development of data pipelines and workflows using Foundry's Pipeline Builder and Code Repositories. Lead integration of external systems and APIs into Foundry for seamless data flow. Work with cross-functional teams to gather requirements, define architecture … TECHNICAL/PERSONNEL SKILLS : Experience designing scalable data models and ontologies. Understanding of semantic data modelling and entity relationships. Familiarity with Foundry's Ontology SDK, Pipeline Builder, and Code Workspaces. Experience integrating external systems and APIs into Foundry. Knowledge of data transformation, cleansing, and enrichment techniques. Awareness of data privacy, access More ❯

days onsite per week End Date: 28/11/2025 Role Overview: We are seeking a highly skilled Hadoop Engineer to support and enhance our Operational Data Platform (ODP) deployed in an on-premises environment. The ideal candidate will have extensive experience in the Hadoop ecosystem, strong programming skills, and a solid understanding of infrastructure-level data analytics. This role focuses on building and maintaining scalable, secure, and high-performance data pipelines within enterprise-grade on-prem systems. Key Responsibilities: Design, develop, and maintain data pipelines using Hadoop technologies in an on-premises infrastructure. Build and optimise workflows using Apache Airflow and Spark Streaming for real-time data processing. … Develop robust data engineering solutions using Python for automation and transformation. Collaborate with infrastructure and analytics teams to support operational data use cases. Monitor and troubleshoot data jobs, ensuring reliability and performance across the platform. Ensure compliance with enterprise security and data governance standards. Required Skills & Experience: Minimum 5 years of experience More ❯

version control systems like Azure DevOps is a strong plus. Main responsibilities: Design, develop, and deploy applications using Google Cloud Functions (Python or Node.js). Implement and optimize data pipelines using Google Cloud Dataflow. Manage and query relational databases using Cloud SQL (PostgreSQL/MySQL). Handle file operations and trigger-based workflows using Google Cloud Storage. Configure More ❯

you the chance to help make the UK a safer place through innovation, working in partnership with government and law enforcement organisations on some of the most challenging data and technology problems out there. We strongly encourage candidates of all different backgrounds and identities to apply. We are committed to building an inclusive, safe and supportive environm ent … Here are some things we've worked on recently that might give you a better sense of the kind of work you could be leading on: Building a data analytics tool using a Neo4J graph database. A DevOps platform to support 3000 users. Processing digital forensic data to enable search and insight generation. A search and … reporting system backed by an ElasticSearch cluster. A custom workflow tool using Camunda, with a React/TS front-end An AI assurance capability Orchestrating a data pipeline and designing specialist metrics for the evaluation of the performance of LLMs in different contexts. Accountabilities: Understands the technical aspects of the project and the wider customer business model. More ❯

Senior Data Engineer UST are currently recruiting for a Senior Data Engineer with proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. About the role The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong … coding skills in Python PySpark and SQL. You will have expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required. Job Description: Work as part of a self-organised Ab Initio … any inconsistencies with the standards (written or best practices) You will be able to estimate the delivery timescales and justify these estimates You will be aware of the Jenkins pipeline principles Knowledge of Agile practices Experience that would be advantageous: You will be aware of AWS interactions and file storage You will have a core understanding of Lambda programming More ❯