101 to 125 of 374 Data Pipeline Jobs in the UK

our award-winning team at Sparta Global. About this role You'll become versatile in a wide array of tools across topics covering data focused coding, data visualisation, Cloud Services, Big Data and Machine learning. You will be equipped to work within a variety … of roles including but not limited to; Data Engineer, Data Analyst, Data Scientist, Information Engineer. You'll be designing, building, maintaining, and troubleshooting the data pipelines that enable organizations to store, process, and analyse their data, and ensuring the data is reliable through testing and debugging. Looking for efficiencies and optimising the data pipelines for scalability and performance will be a focus. You'll be handling and working with large sets of structured and unstructured data and will be responsible for ensuring that the more »

our award-winning team at Sparta Global. About this role You'll become versatile in a wide array of tools across topics covering data focused coding, data visualisation, Cloud Services, Big Data and Machine learning. You will be equipped to work within a variety … of roles including but not limited to; Data Engineer, Data Analyst, Data Scientist, Information Engineer. You'll be designing, building, maintaining, and troubleshooting the data pipelines that enable organizations to store, process, and analyse their data, and ensuring the data is reliable through testing and debugging. Looking for efficiencies and optimising the data pipelines for scalability and performance will be a focus. You'll be handling and working with large sets of structured and unstructured data and will be responsible for ensuring that the more »

A global leading commodities trading firm have partnered with Referment to find a skilled Data Engineer to join their London team on a contract basis. This organisation has some exciting Greenfield projects focussed on modernisation of data warehouse, data lakes and data streaming services. This is critical role, which will require designing, implementing, and enhancing data solutions to support the businesses trading operations. This role will involve building out data pipelines, ETL processes and data warehouses and will require a well-rounded data … engineer with deep experience in Python and other OO languages like Java. Delivery is vital within this role so experience in scaling up data systems using the best data engineering processes is required. Due to the nature of commodities trading, there is exciting problems to solve more »

Specialist/Architect to join our team. The ideal candidate will possess extensive knowledge and hands-on experience with Palantir Foundry, including its data integration, analytics, and operational decision-making capabilities. As a Palantir Foundry Specialist/Architect, you will be responsible for designing, developing, and maintaining data-driven solutions that leverage Palantir Foundry's platform to address complex business challenges. Key Responsibilities: - Design and architect end-to-end data solutions using Palantir Foundry, ensuring alignment with business goals and technical requirements. - Develop scalable and efficient data models, pipelines, and workflows within … the Foundry environment. - Collaborate with stakeholders to understand business needs and translate them into technical requirements and solutions. - **Data Integration:** - Implement data integration solutions to ingest, transform, and manage large datasets from various sources into Palantir Foundry. - Ensure data quality, consistency, and security throughout more »

and Oaktree Capital. Find out more about us by visiting our website at: http://www.hartreepartners.com/JOB OVERVIEW: As a Senior Data Scientist at Hartree, you will play a pivotal role in shaping our data strategy and driving the development of advanced machine learning … time series forecasting solutions, specifically focused on the European Power Markets. You will collaborate with cross-functional teams to design, implement, and optimize data pipelines and predictive models that inform trading decisions and enhance operational efficiency. JOB RESPONSIBILITIES: Market Data Analysis: Conduct comprehensive analysis of data … Python and its data science libraries (e.g., Pandas, NumPy, Scikit-Learn, TensorFlow, PyTorch). Solid understanding of ML and data pipeline architectures and best practices. Experience with big data technologies and distributed computing (e.g., Spark, Hadoop) is a plus. Proficient in SQL and more »

Palantir Data Engineer £85,000 - £95,000 UK Wide Must be able to be eligible for SC clearance! Travel to client site when required! A leading consultancy-based organisation is currently looking to grow their cloud platforms team and wanting to add a Palantir Data Engineer … to their practice. The successful Palantir data engineer will be passionate about cloud technology being involved in using Palantir data focussed reference architecture to assist in designing and building data engineering solutions supporting planning and implementations. Skills and Experience Experience of developing enterprise grade … and deliver the data solutions in Palantir Strong experience with Palantir Data Engineering features such as Code Repo, Code Workbook, Pipeline Builder, migration techniques, Data Connection and Security setup Developing data integration pipelines, transformations, pipeline scheduling, Ontology, and applications in more »

Data Engineer Location: London – Hybrid Employment: Full Time All applicants must hold a DV (Developed Vetting) clearance to apply. The Opportunity: For the Ministry of Defence (MOD), maintaining National Security is a 24-hour-a-day, 365-day-a-year mission. For this reason, it partners with this … Big 4 Firm to support its multidisciplinary AI & Cloud Data Engineering Capabilities. The Role: As an AI & Cloud Data Engineering practitioner, you’ll build large-scale data pipelines and infrastructure, utilising data integration tools to optimise complex data solutions. Collaborating … with high-performing Data Science and ML Engineering teams to drive innovation and insight through AI and Data across the Defence & Security sector. Required Skills/Experience: British Passport Holder with active DV (Developed Vetting Clearance) End-to-end development with data pipelines, ETL more »

consultancy client. The successful candidate will be responsible for designing, developing, and maintaining KDB databases, as well as creating and optimizing real-time data analytics solutions. You will work closely with the technology team to enhance our data management systems, ensure data accuracy and … integrity, and help us make data-driven decisions. Key Responsibilities: Design and develop KDB databases and data processing systems to support real-time data analysis and reporting. Optimize database performance, troubleshoot issues, and implement improvements as needed. Collaborate with data analysts and … other teams to understand data requirements and develop efficient data retrieval processes. Write and maintain KDB Q scripts, ensuring code is well-documented and follows best practices. Monitor data quality and accuracy, implementing data validation and cleansing processes. Develop and maintain data more »

Are you an experienced Data Engineer ? Do you have a passion for Data and experienced with the Microsoft stack ? Do you have SQL, Microsoft Fabric and Azure skills ? If so I have a fantastic new role with my client, with offices in Edinburgh and Glasgow, please … are planning to re-locate. You will be part of a team that works in an Agile environment, contributing to the companies wider data strategy. Responsibilities include: Modernising existing key business solutions by reverse engineering and rearchitecting for Azure. Be involved in designing and implementing systems to effectively … whilst maintaining data integrity and provenance. Experience designing and building robust, production ready, scalable data pipelines Familiarity with optimising strategies, pipeline architectures, data sets and ETL/ELT processes Have experience of selecting the appropriate tools to deliver the solution Coding skills, could more »

Rivington Rd, Preston Brook, Runcorn, WA7 3DJ Salary: £46,000 - £49,000 Hours: 37.5 hours per week Are you a seasoned expert in data engineering looking to make a significant impact in the healthcare industry? Phoenix Medical Supplies is seeking a Senior DataOps Engineer to lead our data … services across the region. As a Senior DataOps Engineer, you will be instrumental in architecting and implementing solutions across the entire data pipeline, from ingestion to visualisation. The role will require you to showcase your skills in data ingestion, storage, modelling, transfer, hosting, visualisation, analysis … and consumption, all while mentoring and training team colleagues. Key Responsibilities: Design and develop robust data pipelines for ingestion, storage, processing, and interface creation. Produce both machine consumable interfaces and human consumable data products, including standard and ad hoc reports. Maintain and optimise data more »

Vertex Search is representing a fast-growing financial services firm that is now looking to make a Senior Data Engineering hire, the first role of its kind into the business with huge scope for impact and influence. The nascent data engineering team will collaborate with internal … teams around the business with a focus on data engineering, pipelines, tooling and services. Key Responsibilities: Data Engineering: Design and develop data services across various business functions Data Pipelines: Design and develop data pipelines for extracting, transforming, and loading data into appropriate storage systems. Data Tools and Services: Develop data tools and services to discover, understand, and utilize data effectively. Best Practice: Advise on and contribute to the development of data governance and management practices. Collaborate across teams to understand more »

on delivering quality services and solutions. Able to distill complex technical challenges to actionable and explainable decisions. Having in-depth knowledge on Azure data components with this feature to provide solutions and necessary recommendations on the architecture. Inspire teams by building consensus and mediating compromises when necessary. Demonstrate … technologies and developing a vision of their suitability and role as a new product within a platform & product landscape. Having knowledge on the data components such as Azure datafactory, data lake, Azure SQL DB, Synapse and visualization with expertise skills in terms of performance tuning and … advanced troubleshooting. 8+ years of experience in data architecture, analytics, design and development Good working knowledge in trouble shooting data warehouse/analytical systems using Azure Synapse. Strong, in-depth experience in data modelling and experience with business intelligence Design and develop batch and more »

Senior Data Engineer (GCP) My client a financial services business are looking for a Senior Data Engineer (GCP), to spearhead global initiatives to migrate complex data systems to GCP. You will design, implement, and optimise data pipelines, ensuring data integrity … during migration and utilising GCP services for scalable and efficient data transformation. Salary: Up to £65K Hybrid: 2 days a week Key Responsibilities: Lead data engineering projects in Europe, working with partners for GCP transition. Design and deploy data pipelines with automated data … GCP. Essential Qualifications: 5+ years of SQL. 3+ years of GCP and its different services. Experience with microservice architecture and data processing pipeline design. Desired Qualifications: Certification in GCP. Financial Services experience. Experience coaching and mentoring data engineers. If this role is of interest, please more »

Are you an experienced Data Engineer ? Do you have a passion for Data and experienced with the Microsoft stack ? Do you have SQL, Microsoft Fabric and Azure skills ? If so I have a fantastic new role with my client, with offices in Edinburgh and Glasgow and … its hybrid working. You will be part of a team that works in an Agile environment, contributing to the companies wider data strategy. Responsibilities include: * Modernising existing key business solutions by reverse engineering and rearchitecting for Azure. * Be involved in designing and implementing systems to effectively handle large … whilst maintaining data integrity and provenance. * Experience designing and building robust, production ready, scalable data pipelines * Familiarity with optimising strategies, pipeline architectures, data sets and ETL/ELT processes * Have experience of selecting the appropriate tools to deliver the solution * Coding skills, could more »

Senior Azure Data Engineer - 6 Month Contract - Inside IR35 This is an Azure Data Engineer 6 month contract position for a financial services end client helping them on delivering and implementing new data pipelines. You will need prior financial services experience as well as … having strong Azure Data Factory and Databricks experience. Work Involved: Implementing and delivering data pipelines Providing mentorship to junior members of the team Working with the Platform team defining capabilities for the lake … framework Developing business conceptual data models Producing new frameworks using design patterns and data ingestion Requirements: Strong Data Pipeline experience Databricks Azure Data Factory Spark Azure cloud Financial Services If you're interested get in touch ASAP more »

Specialist/Architect to join our team. The ideal candidate will possess extensive knowledge and hands-on experience with Palantir Foundry, including its data integration, analytics, and operational decision-making capabilities. As a Palantir Foundry Specialist/Architect, you will be responsible for designing, developing, and maintaining data … driven solutions that leverage Palantir Foundry's platform to address complex business challenges. Key Responsibilities: *Solution Architecture:* - Design and architect end-to-end data solutions using Palantir Foundry, ensuring alignment with business goals and technical requirements. - Develop scalable and efficient data models, pipelines, and workflows within … the Foundry environment. - Collaborate with stakeholders to understand business needs and translate them into technical requirements and solutions. *Data Integration:* - Implement data integration solutions to ingest, transform, and manage large datasets from various sources into Palantir Foundry. - Ensure data quality, consistency, and security throughout more »

Data Engineer – Azure A fast growing and innovative company are looking for a Data Engineer to join their team. They are a people focused company, who since 2021 have won 3 best place to work awards, representing a fantastic working environment. Based in either Derby or … the Chief Operating Officer, the candidate will play a crucial role in driving the company's digital initiatives forward. Key Responsibilities As a Data Engineer, you will play a pivotal role in supporting the delivery and evolution of the company's digital portfolio. Collaborating with a talented team … the candidate will oversee the management and maintenance of existing data pipelines, monitor workloads, conduct root cause analysis, ensure data quality, and escalate technical issues as necessary. What you’ll be getting up to: Monitoring and recovery of Azure Data Factory and Databricks jobs more »

Senior Data Engineer (Remote) - Stealth SaaS Startup Location: Manchester (Remote Friendly) Salary: £65,000 - £75,000 per annum About Us: We are an innovative stealth start-up set to launch a cutting-edge SaaS product next year. We’re looking for a talented Senior Data Engineer to join our team and build robust data pipelines. Responsibilities: Build, and maintain scalable data pipelines. Maintain the pipeline through Databricks and Apache Beam Seamless data integration and quality. Data storage solutions using GCP (will still be considered with … to bring the AI/ML models into production Databricks for daily data processing Qualifications: Experience in data engineering and pipeline construction. GCP and familiarity with other cloud environments. Understanding of SDLC. Experience with Databricks. Programming skills (Python and SQL). What We Offer more »

Job Title: Cloud Data Engineer Location: Knutsford, UK Hybrid work: 2-3 days a week Capgemini is seeking a Cloud Data Engineer t o join Capgemini Financial Services I&D Practice. This is a permanent, full-time position in Knutsford, UK and this represents a unique … opportunity for someone to enhance their I&D Consulting career. What you’ll do and bring: Proficiency in AWS services relevant to data engineering such as S3, Glue, EMR, Athena, and Lambda. Hands-on experience with Snowflake, Redshift cloud data warehousing solutions. Familiarity with DBT (Data … Glue, Starburst, Snowflake, Redshift, Starburst, Hybrid(on-prem to cloud) for building data pipelines and ETL processes. Experience with data pipeline orchestration tools like Apache Airflow . Knowledge of SQL and ability to write complex queries for data transformation and analysis. Understanding of more »

Job Title: Cloud Data Engineer Location: Northampton,UK Hybrid work: 2-3 days a week Capgemini is seeking a Cloud Data Engineer t o join Capgemini Financial Services I&D Practice. This is a permanent, full-time position in Northampton, UK and this represents a unique … opportunity for someone to enhance their I&D Consulting career. What you’ll do and bring: Proficiency in AWS services relevant to data engineering such as S3, Glue, EMR, Athena, and Lambda. Hands-on experience with Snowflake, Redshift cloud data warehousing solutions. Familiarity with DBT (Data … Glue, Starburst, Snowflake, Redshift, Starburst, Hybrid(on-prem to cloud) for building data pipelines and ETL processes. Experience with data pipeline orchestration tools like Apache Airflow . Knowledge of SQL and ability to write complex queries for data transformation and analysis. Understanding of more »

Data Engineer Our client based near Manchester is looking for a skilled Data Engineer to join their agile team. In this role, you'll play a pivotal part in designing, building, and maintaining data pipelines, ensuring the integrity and availability of data for various stakeholders, including BI Teams, Analytics, and Data Scientists. Responsibilities: Design, build, and automate data flow processes and ELTs to provide comprehensive data from business systems. Collaborate with product development teams to ensure best practices and data integrity from … source to destination. Assist in integrating data between systems, including third-party platforms. Write specifications and documentation for data processes. Continuously explore new solutions, tools, and processes to enhance data operations. Requirements: Bachelor's or Master's degree in Computer Science, related field, or more »

Job Title: Data Operations Analyst Company: Global Asset Manager Location: City of London, 3 days per week in office Salary: £85,000 per annum + Benefits About the Company: Join a prestigious global asset manager with over $200 billion in assets under management. Our firm is renowned for … excellence. Based in the heart of London's financial district, you'll be part of a dynamic team shaping the future of financial data operations. Role Overview: As a Data Operations Analyst, you will play a pivotal role in implementing a cutting-edge global platform aimed … at centralizing data management processes. Leveraging your expertise in R, Python, SQL, and Enterprise Data Management (EDM), you'll contribute to streamlining data workflows and optimizing data quality across the organization. Key Responsibilities: Collaborate with cross-functional teams to understand data more »

Data Engineer £80,000-£95,000 London, Hybrid Permanent I am seeking an experienced Data Engineer who is passionate about designing, optimising, and maintaining AI-native data processing pipelines. Your role will involve tackling complex challenges with robust data pipelines across diverse … sources, enhancing the quality of data for our systems, and readiness to meet the challenges of scale. Responsibilities: Ensure the secure handling of data through end-to-end encryption, data isolation, and other best practices. Design robust data processing and indexing pipelines … handling large amounts of data across several modalities, ensuring robustness and timeliness. Curate and manage large-scale data ingestion and indexing pipelines, ensuring data quality and error handling. Manage cloud orchestration and data infrastructure, applying your expertise with various data more »

About the Role: I am looking for a talented Data Engineer to join our clients dynamic team on a 6-month contract basis. This is an exciting opportunity for a mid-level professional with 3-5 years of experience to make a significant impact on our data …/hybrid working options and a competitive day rate of £250-£400, falling inside IR35 regulations. Key Responsibilities: Design, develop, and maintain scalable data pipelines and ETL processes using AWS, Databricks, Python, Spark, and SQL. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions. Optimize and troubleshoot data workflows to ensure efficiency and reliability. Implement data quality and validation checks to ensure data integrity. Contribute to the development and maintenance of our data lake and more »

Data Engineer - London - £75,000 I am working with a hugely successful, global organisation who are looking to add a Data Engineer to their growing London based data team. This organisation have recently implemented a brand new data platform encompassing the full … Microsoft Azure Stack including leveraging Databricks in order to deliver on its goal of becoming truly data driven. In this role you will be responsible for the BI and data engineering lifecycle across the business supporting the global data needs of various business functions. … This role is a split between data engineering and data analytics and you will take a prominent role on a number of exciting projects. You will be joining a business that values its people at the centre of everything they do. They are looking to expand more »