1 to 25 of 492 Data Pipeline Jobs in the UK

member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. Job Title: Data Architect – Azure Databricks/Azure Data Factory For London Market Insurance Exp Required – 10 years Plus Location: London About the Job you are considering … optimisation Unity Catalog, if applicable Databricks SQL, if applicable Cluster sizing and cost optimisation Security and access control patterns Azure Data Factory Pipeline orchestration Linked services and datasets Integration runtime Parameterised pipelines Triggers and scheduling Error handling and retry patterns Metadata-driven ingestion frameworks Monitoring and operational ...

Minimum 10+ years of experience as a Data Engineer or similar role. Proven expertise in the technologies below, and data pipeline development and strong understanding of data warehousing concepts and practices. Excellent problem-solving and analytical skills and strong communication and teamwork skills. … Databricks ML and Azure ML to develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage ...

Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data … proficiency, including experience with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience ...

Principal Data Architect Location: Remote Duration: 6 months £700 - £800 per day - Umbrella only Our client is looking for Principal Data Architect for Data Science family under DSF as part of Professional services sell. This request for NGET under the department of Customer Network … GDPR and Regulatory/CNI control environments In-depth experience in providing resolution to an extensive range of complicated data pipeline related problems, proactively and as issues surface. Demonstrable ability to troubleshoot problems across infrastructure, platform, and application domains. Controlling access to the data ...

Related Jobs Data Engineer Senior Data Engineer Data Engineer Data Engineer Senior Data Engineer - (Python & SQL) Related Jobs Senior Data Engineer (INV) Senior Consultant, Data Engineer, AI&Data, UKI (INV) Senior Consultant, Data … Java, or Scala) Extensive experience with cloud platforms (AWS, GCP, or Azure) Experience with data warehousing and lake architectures ETL/ELT pipeline development SQL and NoSQL databases Distributed computing frameworks (Spark, Kinesis etc) Software development best practices including CI/CD, TDD and version control Strong ...

Senior Data Engineer Remote UK | Occasional Travel Senior Data Engineer | Azure | Microsoft Fabric | Data Platform | Python | SQL | Power BI | AI & Analytics 167 Solutions are partnering with a global organisation embarking on a significant data transformation programme. As investment in analytics … business intelligence continues to grow, we are seeking a Senior Data Engineer to play a key role in shaping and delivering a modern enterprise data platform. This opportunity offers the chance to work on large-scale data initiatives, modern cloud technologies and next-generation ...

solutions for performance, reliability, and scalability, ensuring efficient data processing and storage. Implement and manage orchestration workflows to automate data pipeline scheduling monitoring and error handling. The Requirements As a Data Engineer you will play a crucial role in transforming data … designing and delivering data pipelines using industry-standard tools and cloud technologies. Knowledge of data engineering concepts, data pipeline development, ETL processes, data lake and data warehousing technologies. Proficiency in working with at least one major cloud data ...

contribute to the shaping, validation, and monitoring of data solutions. Your work ensures accuracy, clarity, and integrity across the entire delivery pipeline RESPONSIBILITIES Data Analysis & Insight Development Exploratory Analytics: Conduct detailed exploratory data analysis to identify trends, root causes, and opportunities for improvement … Insight Packaging: Translate complex findings into clear, concise insights, visualisations, and recommendations for both technical and non-technical stakeholders. 2. Data Preparation & Pipeline Support Data Wrangling: Prepare, clean, and transform datasets to support modelling, prototyping, and production-ready pipelines. Requirements Translation: Work with the Data ...

both our Customers and Employees. So, what are you waiting for? Join a community that cares about you! More about the team The Data and Platform Engineering team are the foundation for the Data Office function. Responsible for designing, building, and maintaining PfP's data … requirements. Data Product Design Standards: Ensuring that data products are discoverable, interoperable, and trustworthy, adhering tohigh standards. Data Pipeline Design: Designing robust data pipelines, including batch, event-driven, and streaming architectures. Operational Considerations: Addressing operational concerns such as security, scalability, performance ...

platforms so they are reliable, secure, and easy to change. You will work across engineering and operations to automate delivery, improve data pipeline observability, and embed good governance so analytics and AI workloads can run at scale. You will be part of the Data Platforms … restricted environments. Delivering data applications in an automated and continuous fashion. Utilising tools like Airflow for orchestration and automating data pipeline deployments using DevOps tools and techniques (Docker, Containers, Terraform, Kubernetes) Building data pipelines using both cloud-native and open-source data ...

About the role... Are you an experienced Data Architect looking for your next challenge? Cooper Parry is on a rapid growth trajectory, and we’re seeking a Data Architect to lead the design and evolution of our enterprise data platform. This role will play … experience with Microsoft Fabric Deep understanding of: Modern data lakehouse architectures Dimensional and analytical data modelling Data pipeline orchestration Strong SQL skills and experience with distributed data processing Experience designing enterprise‐scale analytics platforms in Azure Ability to translate business requirements ...

Senior Data Engineer 6 Month contract with extension Full-time Inside IR35 Hybrid 1 day every two weeks in the office Nottingham - NG1 4BX Join our client in embarking on an ambitious data transformation journey using Databricks, guided by best practice data governance … warehousing. Able to write repeatable, production-quality code for data pipelines, utilizing templating and parameterization where needed. Can make data pipeline design recommendations based on business requirements. Experience with data migration is a plus. Open to new ways of working and new technologies. ...

largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. Who You Will Be Working With The Data Platforms team is part of the Insights and Data Global Practice and has seen strong growth and continued success across a variety … projects and sectors. Data Platforms is the home of the Data Engineers, Platform Engineers, Solutions Architects and Business Analysts who are focused on driving our customers digital and data transformation journey using the modern cloud platforms. We specialise on using the latest frameworks, reference ...

Department: IT Type: 3-month rolling contract Outside IR35 Position: Analytics Engineer The organisation is seeking an experienced Analytics Engineer to join its Data & Analytics team. The role focuses on building scalable data pipelines, transforming raw data into clean, reusable datasets, and enabling data-driven decision-making. Key Responsibilities Design and build data products, with proficiency throughout the data lifecycle. Develop robust data models through close collaboration with business users and the engineering team. Partner with senior management, operational leads, and other stakeholders, coaching and supporting ...

largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. Who You'll Be Working With As a Data Solution Architect at Capgemini, you will design and lead the delivery of modern data platforms that enable analytics, AI, GenAI, and data … Fabric experiences to operationalise models and drive business insights. Design, monitor and optimise data pipelines and platform performance: Use Fabric capacity management, pipeline monitoring, logging, and operational best practices to improve reliability, throughput, cost, and scalability. Collaborate with cross-functional teams: Work closely with business analysts, data ...

Lead Data Engineer Location: London, City (Hybrid) £ Highly competitive base with bonus and benefits Lead Data Engineer is sought by expanding central data platform team within a prestigious global wealth management organisation. This role offers the opportunity to help shape enterprise-wide data engineering standards, contribute to platform strategy, and build modern, scalable data solutions that support business-critical decision-making. Essential to this role are strong python skills, expertise in MS Fabric, SQL, data warehousing, Data Lake and Azure technologies. The Role ...

identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/… concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/GOV.UK/Public Sector domain experience strongly preferred. Must be active SC Cleared or fully eligible ...

identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/… concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must be active SC Cleared or fully ...

Data Architect £Up to £120,000 GBP Competitive Bonus + Car Allowance Hybrid WORKING Location: London; Bristol; Manchester, Central London, Greater London - United Kingdom Type: Permanent Data Architect London | Manchester | Bristol | Hybrid | Permanent Our client is a leading global technology and consulting organisation , recognised … environments. Due to the nature of these programmes, active SC clearance is required for this role . This opportunity sits within a growing Data & AI Architecture capability , focused on delivering complex, high-impact programmes at the heart of government. The role combines deep technical architecture expertise with leadership ...

Data Architect £Up to £120,000 GBP Competitive Bonus + Car Allowance Hybrid WORKING Location: London; Bristol; Manchester, Central London, Greater London - United Kingdom Type: Permanent Data Architect London | Manchester | Bristol | Hybrid | Permanent Our client is a leading global technology and consulting organisation , recognised … environments. Due to the nature of these programmes, active SC clearance is required for this role . This opportunity sits within a growing Data & AI Architecture capability , focused on delivering complex, high-impact programmes at the heart of government. The role combines deep technical architecture expertise with leadership ...

Data Architect £Up to £120,000 GBP Competitive Bonus + Car Allowance Hybrid WORKING Location: London; Bristol; Manchester, Central London, Greater London - United Kingdom Type: Permanent Data Architect London | Manchester | Bristol | Hybrid | Permanent Our client is a leading global technology and consulting organisation , recognised … environments. Due to the nature of these programmes, active SC clearance is required for this role . This opportunity sits within a growing Data & AI Architecture capability , focused on delivering complex, high-impact programmes at the heart of government. The role combines deep technical architecture expertise with leadership ...

largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. Who You'll Be Working With As a Data Solution Architect at Capgemini, you will design and lead the delivery of modern data platforms that enable analytics, AI, GenAI, and data … ready architectures, taking end to end ownership from vision through to delivery and value realisation You will be part of the Data Platforms team that is part of the Insights and Data Global Practice that has seen strong growth and continued success across a variety ...

seeking an experienced Data Architect to join our Group Technology team on a 6-month contract. You will help realise the value of the immense Connells Group data landscape for the benefit of customers, colleagues and the overall business. You will use your expertise to define … evolve and communicate the existing Estate Agency enterprise data architecture, to democratise safe access to consolidated and unified data, modelled to help engineering teams develop single customer and property views. The data architecture will also support system migration and integration, to establish effective master ...

Data Platform ManagerUK/RemoteCirca £65k - £75k per annum + bonus & benefits Are you ready to take ownership of a data platform and shape how data drives business decisions? This is a rare opportunity to step into a leadership role where you'll combine … hands-on data engineering with strategic product management, all while building and mentoring a high-performing team. You'll play a pivotal role in evolving a modern data platform (Microsoft Fabric/Azure Synapse), ensuring it delivers real business value, scalability, and innovation. ...

Data Engineer Peterborough (Hybrid) £50,000 - £60,000 Base + Benefits We are seeking an experienced Data Engineer with strong Databricks experience to join a growing Data & Analytics team. As part of this continued growth, the successful candidate will play a key role … developing and enhancing data solutions, working closely with the Data & Analytics Manager and wider team to support the delivery of data-driven initiatives. This is an excellent opportunity for a skilled Data Engineer looking to make an impact within a forward-thinking ...