151 to 175 of 381 Data Pipeline Jobs in the UK

Job Overview Position: Senior Data Engineer In this role, you will be responsible for storing, processing, modeling, and applying data science to ensure that data and insights are available for data analytics and business intelligence (BI) systems. You must be proficient in … data assessment and profiling, source system analysis, and providing a clear vision for processing and interpreting data to support its end-use. Ideal Candidate Profile The ideal candidate will have a background in Data … Engineering or Data Warehousing (Data Engineer/DW Developer) with robust DW development and ETL/ELT data pipeline experience, along with expert knowledge of SQL. Key Skills - Mandatory Strong data modeling and SQL/database design skills Proficiency in ETL more »

As a Senior Data Engineer, you will play a central role in shaping our data infrastructure and operations. Your primary focus will be on establishing a robust foundation that is sustainable for future growth. You'll be responsible for collecting data from various sources … ensuring its structure, and making it easily accessible for Business Intelligence and Data Science purposes. Key Responsibilities: Data Infrastructure Ownership : Take ownership of the end-to-end data infrastructure, from aggregating data to designing, building, and maintaining scalable and efficient pipelines. Cross … Cloud Platform (GCP) is highly preferred (experience with other cloud platforms like AWS or Azure is also considered). Familiarity with data pipeline scheduling tools like Apache Airflow. Ability to design, build, and maintain data pipelines for efficient data flow and processing. Understanding more »

Data Engineer Consultant - Data & AI (DV Cleared/Eligible)65,000 – 75,000 + packageLocation: United Kingdom (London – London) Type: PermanentData Engineer Consultant - Data & AI (DV Cleared/Eligible)Location: LondonSalary: 65k - 75kAbout the Role:Our client, a leading consultancy in the Defence & Security … sector, is seeking a skilled Data Engineer to join their AI & Data Engineering team. This pivotal role offers the opportunity to be at the forefront of advanced technology projects aimed at enhancing national security. The position is perfect for individuals who are passionate about leveraging their … data security and quality across projects.Qualifications and Skills:Proven experience in data engineering, particularly with ETL processes, data pipeline construction, and workflow orchestration.Familiarity with distributed computing techniques, such as parallel processing and batch data management.Strong foundation in data system more »

Data Engineer Consultant - Data & AI (DV Cleared/Eligible)65,000 – 75,000 + packageLocation: United Kingdom (Bristol – Bristol) Type: PermanentData Engineer Consultant - Data & AI (DV Cleared/Eligible)Location: BristolSalary: 65k - 75kAbout the Role:Our client, a leading consultancy in the Defence & Security … sector, is seeking a skilled Data Engineer to join their AI & Data Engineering team. This pivotal role offers the opportunity to be at the forefront of advanced technology projects aimed at enhancing national security. The position is perfect for individuals who are passionate about leveraging their … data security and quality across projects.Qualifications and Skills:Proven experience in data engineering, particularly with ETL processes, data pipeline construction, and workflow orchestration.Familiarity with distributed computing techniques, such as parallel processing and batch data management.Strong foundation in data system more »

Data Engineer Consultant - Data & AI (DV Cleared/Eligible)65,000 – 75,000 + packageLocation: United Kingdom (Greater Manchester – Manchester, Greater Manchester) Type: PermanentData Engineer Consultant - Data & AI (DV Cleared/Eligible)Location: ManchesterSalary: 65k - 75kAbout the Role:Our client, a leading consultancy in … the Defence & Security sector, is seeking a skilled Data Engineer to join their AI & Data Engineering team. This pivotal role offers the opportunity to be at the forefront of advanced technology projects aimed at enhancing national security. The position is perfect for individuals who are passionate … data security and quality across projects.Qualifications and Skills:Proven experience in data engineering, particularly with ETL processes, data pipeline construction, and workflow orchestration.Familiarity with distributed computing techniques, such as parallel processing and batch data management.Strong foundation in data system more »

Data Engineer We are seeking a Data Engineer to join our friendly team. Our client is building a new data platform centred around a Microsoft SQL server-based data warehouse that runs on AWS RDS. This role is crucial in expanding our … client's data capabilities and supporting the launch of our new platform and mobile app. Key Responsibilities: Support and maintain the central Data Platform, ensuring its availability and performance to meet business needs. Build data pipelines to connect APIs, files, databases, and data streams to our client's Data Platforms. Shape data from various sources into a transactional ODS for detailed reporting and a Kimball-modelled Data Warehouse for analytics and dashboards. Define and embed data quality measures within data pipelines. more »

About Us: iOpt Ltd is a IoT technology company using sensors to capture property data to help build smart solutions for sustainable and efficient living. The company works with a range of clients across different sectors. We are an innovative and rapidly growing company and we now require … a talented and motivated Data Analyst to join our team and contribute to our exciting journey. Job Purpose This role has been created to support the growth in our Data team in iOpt. You will be responsible for making sure we have a strong understanding of … Project Management: Apply knowledge and experience of project management methodologies. Utilize appropriate tools and techniques to contribute effectively to project success. Data Pipeline Development: Design, build, test, and deploy data pipelines to support analytic products. Ensure the efficiency and reliability of data flows more »

We are seeking an experienced Python Data Engineer Consultant to join a leading hedge fund. The ideal candidate should have a proven ability to design and implement data solutions, with expertise in extracting, transforming, and loading (ETL) data from various sources, including websites, PDFs … and Real Time data streams. Role: Python Data Engineer Client: Leading Hedge Fund Type of role: contract (6 months) Location: London office Rate: up to £1500/day Key Responsibilities: Architect, develop, and optimize data pipelines for extracting, transforming, and loading data from diverse alternative data sources, such as web scrapes, PDF files, and Real Time data infrastructures. Implement and manage database solutions in SQL Server and various cloud-based environments, ensuring data integrity, efficiency, and accessibility. Build and maintain Real Time data more »

Senior Data Engineer Salary Negotiable to £70,000 DoE Hybrid working with 2 days in London Job Reference J12785 Work with an award-winning independent protection adviser. Develop the data engineering capabilities. Data is a key enabler for the business and developing enterprise grade … data engineering services is a central pillar in the digital transformation program. The data pipelines and models that you will develop will underpin all Business Intelligence, Machine Learning and CRM capabilities. The role holder must ensure that the data is well-managed and ready … for consumption by the business, which will require the integration of multiple data sources (both cloud and on-prem) and developing data transformations pipelines in an Azure Synapse environment. The role will be aligned to a specific area of the business e.g. the digital teams or more »

understand client requirements and provide technical expertise during pre-sales engagements. Design and implement Proof of Concepts (POCs) to showcase the potential of data integration solutions and demonstrate value to clients. Prepare technical proposals and cost estimates for data integration projects. Conduct client presentations and answer … technical questions effectively. Design, develop, and deploy data pipelines for ingesting, transforming, and loading data from various sources into target systems (cloud and on-premises). Implement data cleansing, quality checks, and transformation logic to ensure data integrity and … accuracy. Configure and manage data integration tools, including cloud-based platforms and on-premises solutions. Automate workflows and schedule data pipeline execution for efficient data processing. Monitor data pipelines for performance and troubleshoot any issues. Collaborate with data analysts more »

business critical reporting systems across the Protein Works including marketing, trading, operations and finance. You will have a strong understanding and appreciation of data engineering concepts and how to apply them using the Google Cloud Platform. You will possess an unwavering passion for data and how … Engineer Immediate Responsibilities Managing and delivering end to end BI reporting projects from initial brief/requirement gathering to data pipeline development (APIs/ETL/ELT/SQL) to visualization Undertaking ad hoc BI tasks based on business requirements and priorities As part of … Coding experience with Python/Javascript/C# Familiarity with statistical/machine learning/AI concepts and techniques Understanding of data pipeline/orchestration tools e.g. dbt, dataform Appreciation of GCP’s serverless technologies e.g. Cloud Run/Workflows Understanding of Google’s marketing stack, Google more »

experienced AWS Cloud Architect to design AWS reference architecture for our services-led client. Responsibilities: Design and implement scalable, secure, and high-performance data architectures on AWS. Develop data models and database designs that align with business requirements and ensure data integrity. Proficiency in … or AWS CodePipeline. Hands-on experience with Infrastructure as Code (IaC) tools like AWS CloudFormation, Terraform, or similar. Create and maintain ETL processes, data pipelines, and data integration workflows using AWS services. Implement data storage solutions, including data lakes, data warehouses, and operational databases. Provide technical leadership and guidance on AWS data services and architecture to the team. Requirements: Proven experience as an AWS Data Architect or similar role, with a strong focus on data architecture and management. In-depth knowledge of more »

Job Title: Data Engineer Location: London, Manchester, or Oxford (Remote) Salary: Competitive + Discretionary Company Bonus Scheme Hours: Monday – Friday (37.5 hours per week) Perks: 25 days holidays (rising to 28 days) plus bank holidays Hybrid working – flexibility between office and remote work Mental health care – 6 free … their families Pension contribution of 5% Free breakfast, drinks, and fruit in the office Employee discounts and volunteer opportunities The Role: As a Data Engineer for our client, you will play a crucial role in building, testing, and maintaining their data platform. This platform serves the … critical needs of various departments including analytics, data science, operations, and marketing. You will be instrumental in designing and maintaining their data stack, including data pipelines and data warehouse. They're currently migrating to a new Data Warehouse based on more »

Data Engineer Location: London, 3 days per week in the office Salary range: £60,000-£65,000 + benefits Purpose: Design, build, and maintain scalable data architectures, including pipelines and cloud-based data warehouses. Tech: Python (NumPy, Pandas), SQL, ETL, Cloud (AWS, Azure or … consultancy who solely specialise in the Maritime trading industry. You will work onsite with 1 specific client to design, build and maintain scalable data architectures working with key stakeholders in their business to shape their data-driven strategies. The Company: They have a long history providing … ETL Processes: Deep expertise in developing and optimising ETL pipelines. Version Control: Experience with Git for code collaboration and change tracking. Data Pipeline Tools: Proficiency with tools such as Apache Airflow. Cloud Platforms: Familiarity with AWS, Azure, Snowflake, and GCP. Visualisation: Tableau or PowerBI Delivery Tools: Familiarity more »

We Want We are looking for a Staff Backend Engineer to join our Insights Back-End team to help build out data pipeline, data store and API layer of Bazaarvoice data that will power our various client-facing dashboards, raw data reporting, product performance reporting. The ideal candidate has built and operated systems that aggregate and share data sets to support new product offerings and is well experienced in evaluating the right technologies to build those data sets. Success in the role is underpinned by … a strong understanding of the business cases of the product portfolio and building relevant data sets, API layer and access methodologies that support the portfolio goals. What You’ll Be Doing Own, design, implement and support of major components of the unified back end data. Define how data more »

Complexio is Foundational AI. This works to automate business activities by ingesting whole company data – both structured and unstructured – and making sense of it. Using proprietary models and algorithms, Complexio forms a deep understanding of how humans are interacting and using it. Automation can then replicate and improve … Management, C Transport Maritime, Trans Sea Transport and BW Epic Kosan. Key Responsibilities 👥 Collaborate with cross-functional teams to design and build scalable data solutions that support product innovations and business insights. 🔨 Develop and maintain robust data pipelines that support complex data collection, ingestion … engineering or a related field. Proficient in and experience with relational databases (e.g., MySQL, PostgreSQL) as well as Strong experience with data pipeline and workflow management tools (e.g. sbt, Nifi). Familiarity with big data technologies, such as Spark. Especially when to use or not. more »

Australia. It’s official: Bazaarvoice is a Great Place to Work in the US , Australia, India, Lithuania, France, Germany and the UK! Staff Data Engineer (P7) - Global Data Platform (Enablement) team What is the BV Data team at Bazaarvoice? Authentic data is … the lifeblood of the Bazaarvoice business. It is imperative that our data is consumable, interpretable, and instantly available to fuel insights across our brand, retailer and consumer customers. The BV Data team works closely with Product, Engineering, Business Development, and Account teams to consistently find new … ways to use data across the business. Who We Want Are you ready to combine your talent for crafting solid data systems and enthusiasm for cutting-edge technology to harness the power of data at Bazaarvoice? We’re looking for a strong data more »

Data Engineer required for a 6 month initial contract with a leading commodities trading firm. Outside of IR35 Hybrid working - 2/3 days in London office Experience Required: - Commodities industry … experience Weather forecasting Airflow (or equivalent data orchestration platforms) Data Modeling (Star Schema) Data Warehousing Data Pipeline Orchestration (Kafka) On-Prem SQL more »

Working with a top tier Digital Asset Management firm who are looking for an adept and experienced Data Engineer to play a critical role in shaping and managing their data engineering platform. This role demands strong technical skills and a proven ability to work closely with … quantitative researchers, ensuring seamless integration of data engineering processes to support our quantitative investment strategies. Key Responsibilities: Collaborate closely with quantitative researchers to understand their data needs, offer technical solutions, and effectively integrate data engineering with quantitative strategies. Design, develop, and optimize robust data pipelines, facilitating ETL processes that support quantitative research, analysis, alpha forecasting, and execution. Implement stringent data quality assurance measures to ensure the accuracy, reliability, and consistency of data inputs and outputs. Evaluate, select, and manage appropriate data engineering tools, technologies, and infrastructure more »

Lead Data Engineer (Contract/ASAP Start) Location| Workplace policy: Fully Remote (Must be based in the UK without visa/sponsorship requirements ) Day Rate: up to £600 (outside IR35) Contract Length: 6-month term initially An exciting market-leading tech-savvy organization that works directly alongside some … of the biggest brands in the retail sector is seeking a contract Lead Data Engineer with hands-on experience working directly with C-Suite execs. Technical Expertise Needed At least 4 years recent Databricks (must have) At least 10+ years experience in a Lead Data Engineer … Educated to degree level in a QS top 100 university Proven experience delivering scalable data pipelines PySpark SQLDevOps/DataOps/CICD Expertise in designing, constructing, administering, and maintaining data warehouses and data lakes Data Modelling/Data Architecture Data more »

A global leading commodities trading firm have partnered with Referment to find a skilled Data Engineer to join their London team on a contract basis. This organisation has some exciting Greenfield projects focussed on modernisation of data warehouse, data lakes and data streaming services. This is critical role, which will require designing, implementing, and enhancing data solutions to support the businesses trading operations. This role will involve building out data pipelines, ETL processes and data warehouses and will require a well-rounded data … engineer with deep experience in Python and other OO languages like Java. Delivery is vital within this role so experience in scaling up data systems using the best data engineering processes is required. Due to the nature of commodities trading, there is exciting problems to solve more »

Specialist/Architect to join our team. The ideal candidate will possess extensive knowledge and hands-on experience with Palantir Foundry, including its data integration, analytics, and operational decision-making capabilities. As a Palantir Foundry Specialist/Architect, you will be responsible for designing, developing, and maintaining data-driven solutions that leverage Palantir Foundry's platform to address complex business challenges. Key Responsibilities: - Design and architect end-to-end data solutions using Palantir Foundry, ensuring alignment with business goals and technical requirements. - Develop scalable and efficient data models, pipelines, and workflows within … the Foundry environment. - Collaborate with stakeholders to understand business needs and translate them into technical requirements and solutions. - **Data Integration:** - Implement data integration solutions to ingest, transform, and manage large datasets from various sources into Palantir Foundry. - Ensure data quality, consistency, and security throughout more »

Are you an experienced Data Engineer ? Do you have a passion for Data and experienced with the Microsoft stack ? Do you have SQL, Microsoft Fabric and Azure skills ? If so I have a fantastic new role with my client, with offices in Edinburgh and Glasgow, please … are planning to re-locate. You will be part of a team that works in an Agile environment, contributing to the companies wider data strategy. Responsibilities include: Modernising existing key business solutions by reverse engineering and rearchitecting for Azure. Be involved in designing and implementing systems to effectively … whilst maintaining data integrity and provenance. Experience designing and building robust, production ready, scalable data pipelines Familiarity with optimising strategies, pipeline architectures, data sets and ETL/ELT processes Have experience of selecting the appropriate tools to deliver the solution Coding skills, could more »

on delivering quality services and solutions. Able to distill complex technical challenges to actionable and explainable decisions. Having in-depth knowledge on Azure data components with this feature to provide solutions and necessary recommendations on the architecture. Inspire teams by building consensus and mediating compromises when necessary. Demonstrate … technologies and developing a vision of their suitability and role as a new product within a platform & product landscape. Having knowledge on the data components such as Azure datafactory, data lake, Azure SQL DB, Synapse and visualization with expertise skills in terms of performance tuning and … advanced troubleshooting. 8+ years of experience in data architecture, analytics, design and development Good working knowledge in trouble shooting data warehouse/analytical systems using Azure Synapse. Strong, in-depth experience in data modelling and experience with business intelligence Design and develop batch and more »

Data Engineer £80,000-£95,000 London, Hybrid Permanent I am seeking an experienced Data Engineer who is passionate about designing, optimising, and maintaining AI-native data processing pipelines. Your role will involve tackling complex challenges with robust data pipelines across diverse … sources, enhancing the quality of data for our systems, and readiness to meet the challenges of scale. Responsibilities: Ensure the secure handling of data through end-to-end encryption, data isolation, and other best practices. Design robust data processing and indexing pipelines … handling large amounts of data across several modalities, ensuring robustness and timeliness. Curate and manage large-scale data ingestion and indexing pipelines, ensuring data quality and error handling. Manage cloud orchestration and data infrastructure, applying your expertise with various data more »