18 of 18 Data Pipeline Jobs in Central London

Data Engineer | London Market Broker | London/Hybrid | £50 - 55,000 (doe) Are you passionate about designing and optimizing data infrastructures? Do you have a flair for working with large datasets and cloud technologies? We are seeking a highly skilled Data Engineer to join … our client, who are a boutique London market broker. As a Data Engineer, you will play a vital role in developing scalable data pipelines, ensuring data quality, and enabling efficient data processing to support business intelligence and advanced analytics. You will also … be involved in their Cloud migration project. Some key responsibilities include: Design, develop, and maintain scalable and reliable data pipelines. Build and optimize data architectures, data lakes, and data warehouses. Develop ETL/ELT processes to extract, transform, and load data More ❯

Data Engineer | London Market Broker | London/Hybrid | £50 - 55,000 (doe) Are you passionate about designing and optimizing data infrastructures? Do you have a flair for working with large datasets and cloud technologies? We are seeking a highly skilled Data Engineer to join … our client, who are a boutique London market broker. As a Data Engineer, you will play a vital role in developing scalable data pipelines, ensuring data quality, and enabling efficient data processing to support business intelligence and advanced analytics. You will also … be involved in their Cloud migration project. Some key responsibilities include: Design, develop, and maintain scalable and reliable data pipelines. Build and optimize data architectures, data lakes, and data warehouses. Develop ETL/ELT processes to extract, transform, and load data More ❯

Job Title: Senior Generative AI Data Engineer ? Location: London, United Kingdom ? Salary: Negotiable ? Recruitment Agency: 167 Solutions ? Job Description: 167 Solutions is seeking an experienced Senior Generative AI Data Engineer to join our client's innovative team in London. This pivotal role offers … the opportunity to lead the development and deployment of generative AI models, driving transformative solutions across various business domains. ? Key Responsibilities: Data Pipeline Architecture: Design, develop, and maintain robust, scalable data pipelines to support generative AI models, ensuring efficient data flow and processing. … or similar. ? Data Processing: Expertise in big data technologies such as Apache Spark, Hadoop, and experience with data pipeline tools like Apache Airflow. ? Cloud Platforms: Strong experience with cloud services, particularly AWS, Azure, or Google Cloud Platform, including services like S3, EC2, Lambda More ❯

The Data Engineer Manager is responsible drive the design, development, and optimization of data solutions in clients data infrastructure. In addition to fostering the growth of a skilled team, you will play a pivotal role in delivering clients data applications, infrastructure, and … and lot's more. You must have experience with Python/Pyspark, Azure and managing a small team of Mid level/Senior Data Engineers. As part of the Technology Hub within the client, Data Engineer Manager will work very closely with all teams across the … client. The role is instrumental in defining and upholding a clear vision for the integrity of data life cycle management aligning with clients strategic goal of becoming a centre of expertise. Additionally, it ensures stewardship of clients data and technical architecture, fostering innovation and reliability across More ❯

Data Production Engineer Location: London, United Kingdom Salary: Competitive + Excellent Benefits Our client, a leading global trading firm, is seeking a talented Data Production Engineer to join their highly collaborative Data team. Data is central to their success, powering one of … the unique opportunity to work directly with live trading teams, automate processes, explore vast datasets, and engage with key external stakeholders such as data vendors, brokers, and exchanges. You'll play a hands-on role in acquiring, validating, and preparing data that feeds cutting-edge quantitative … Comfortable using the Linux command line for file manipulation, automation, and system monitoring. Experience with financial datasets (e.g. Refinitiv, S&P, Bloomberg) and ETL pipeline management is highly desirable. Prior exposure to supporting systems in a production trading environment is a strong advantage. Why Apply? You'll join a More ❯

Data Lead - Strategy & Integration - £500 (outside IR35) - London/Hybrid Investigo are supporting a growing IT Consultancy, currently going through an exciting period of change due to acquisitions, and now require a strategic, hands-on Data Strategy & Integration Lead. This hire will help architect and execute … the company-wide data infrastructure from the ground up. Building the data warehouse, connecting disparate systems, and driving the overall data vision. £500 p/day (Outside IR35) 6 months (with view to extend) Hybrid - London (city)/remote. The team currently work … systems and environments is essential Deep experience integrating CRM and financial platforms Solid understanding of ETL/ELT processes and modern data pipeline tools Hands-on experience managing data migrations from diverse legacy systems into new consolidated environments (e.g., post-M&A or system upgrades More ❯

Data Engineer 4 months - with extensions Remote Active SC clearance required £640 per day inside ir35 REQUIRED Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc Strong understanding of complex JSON manipulation Experience working … with Data Pipelines using a custom Python/PySpark frameworks Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data. Strong understanding of Data Security principles - data owners, access … CLI usage and scripting) Git Markdown Scala DESIRABLE Azure SQL Server as a HIVE Metastore DESIRABLE TECHNOLOGIES Azure Databricks Apache Spark Delta Tables Data processing with Python PowerBI (Integration/Data Ingestion) JIRA If you meet the above requirements, please apply for the vacancy to be More ❯

We are currently recruiting for a Data Engineer with hands-on experience of Azure and Insurance experience to join one of our Global Insurance Clients on a 6-month contract. Please note, this role is Inside IR35 with travel to London 1 day a week (Mondays). Key … Responsibilities: Design, build, and maintain robust data pipelines to curate and ingest data efficiently. Work with Azure cloud services to deploy and manage scalable data solutions. Implement and maintain CI/CD pipelines for seamless data workflows. Collaborate with cross-functional teams … to ensure data integrity, security, and accessibility. Key Skills & Experience: Strong programming skills in Python (PySpark). Hands-on experience with Azure Data Services (Azure Data Factory, Databricks, Synapse, Data Lakes, etc.). Experience with CI/CD pipelines for release management. More ❯

A leading Global Technology Consultancy in the City of London is now looking for a Lead Data Engineer to join the business on a permanent basis. The role is offering a base of £110,000 (potential flexibility) + bonus and shares. Their HQ is in the City of … London and they have adopted a hybrid working model. The ideal Lead Data Engineer will have a background from either an IT Technology provider or IT Consultancy and enjoys working with clients to solve real world data problems. In this exciting role, you'll partner with … clients to transform raw data into powerful insights, ensuring top-tier performance, reliability, and compliance with data governance standards. The Lead Data Engineer will be well versed in the latest Data Tools/Gen AI and can utilise them to delivery high More ❯

Join a leading force in the music industry, empowering artists at every stage of their careers. We're looking for a Lead Data Engineer to spearhead the development and optimisation of our cutting-edge Analytic Data Platform (ADP). You’ll design scalable data solutions, lead a talented team of engineers, and collaborate across tech and business teams to deliver impactful data products. Your work will directly support strategic decision-making and innovation across the company. What you’ll do: Architect and build efficient, reliable data pipelines and … products. Lead and mentor a growing data engineering team. Collaborate with DevOps, QA, IT Ops, analysts, and business experts. Champion best practices in data design, development, and cloud architecture. Stay on top of the latest tools and trends, especially in AWS and open-source data More ❯

+ bonus + benefits Experience: Strong ML Engineering/MLOps Deploying models to production Monitoring model performance Retraining models Data pipelines Python, Kubernetes T erraform SQL Cloud Azure/AWS or GCP Desirable infrastructure; Windows/Linux etc More ❯

party vendors (enterprise software solutions such Document Management, Time Management, Practice Management, Core business i.e. Microsoft etc) to support their integration and master data management implementations Develop Azure-based integration solutions using Logic Apps, Service Bus, Function Apps, API Management, and Data Factory (ADF). Build … robust data pipelines and ETL solutions that support business processes and reporting needs. Conduct code reviews on other people's solutions Working within a team of architects, data engineers, data analysts and C#/.Net Core developers the Integration Developer/Azure Integration Developer … optimises their technology environment. As such you will bring a mix of skills including: Expertise in Azure Integration Services suite with strong Azure Data Factory ETL Logic Apps, Service Bus, API Management Very strong SQL skills, including database design, stored procedures, and query optimisation Underlying programming expertise in More ❯

also key, you will be leading from the front and setting standards. The environment is still on-prem and although they have some data pipelines it's mainly a legacy environment. They promote Agile and DevOps ways of working so any experience driving these would be a great More ❯

also key, you will be leading from the front and setting standards. The environment is still on-prem and although they have some data pipelines it's mainly a legacy environment. They promote Agile and DevOps ways of working so any experience driving these would be a great More ❯

create modern systems by using Microsoft Fabric Act as a consultant, understanding the client requirements and deliver the best solutions. Building and optimising data pipelines. Debugging issues Keep costs under control Skills Microsoft Fabric - Lakehouse and warehouse models Azure Tools (Data Factory, Synapse, SQL) Python experience … for programming Databases, SQL and NoSQL (CosmosDB, KQL) Data Modelling: Kimball frameworks and 3NF Nice To Have Databricks Power BI AI/ML Azure Infrastructure DevOps Certifications Microsfot certified: fabric Analytics Engineer Associate DP-203 Azure data Engineering AZ-305 Azure Solutions Architect. If this sounds More ❯

Leading Financial Services client is currently going through a large Data transformation and is looking for Data Solution Manager to join them on this exciting journey. The roles are offering a base between £80,0000 to £85,000pa + a strong benefits package and flexible working. … One day in office in every 3 months in East Sussex As a Data Solution Manager you will report to our Head of Data Delivery and lead a small team of some of the best engineers in the industry to deliver, improve and maintain robust data pipelines within our ambitious data architecture. Requirements: You will have a strong background in data engineering, with experience in managing and mentoring other engineers. You should have significant experience with Snowflake and Microsoft Azure, particularly Azure Data Factory and Azure Dev Ops. More ❯

of experienced Clojure engineers responsible for a range of systems including: Page composition and rendering for article and channel pages A configuration platform Data pipelines pushing Oracle DB content into Elasticsearch Internal jobs and feeds systems Sitemaps Elements of a comments service These are monolithic, on-premise backend More ❯

wed love to hear from you. What Youll Do Design, build, and maintain Python services that power critical financial workflows. Work with modern data processing tools like pandas and polars to transform and analyze large datasets. Build async Python services using asyncio , and develop event-driven systems integrating … speed with long-term maintainability. Work closely with product and business stakeholders to deliver high-impact features. Support and evolve our PostgreSQL -based data storage and querying layers. What Were Looking For 36+ years of professional software development experience, primarily with Python . Solid experience with Pythons standard … library and familiarity with modern idioms (e.g., dataclasses , type annotations , async/await ). Experience with data transformation and analysis libraries like pandas or polars. Comfortable writing unit and integration tests with pytest . Practical experience with PostgreSQL (or another relational database), and understanding of schema design and More ❯