301 to 325 of 363 Data Pipeline Jobs in Central London

Position: Data Engineer Location: Central London or Bristol (3 days per week onsite) Salary: £600 to £650 p.d inside IR35 Security Clearance: SC Clearance Required About the Role A leading IT consultancy delivering innovative solutions to the UK public sector is seeking an experienced Data Engineer to support a high-profile programme. You'll collaborate with … cross-functional teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation … Docker/Kubernetes) - experience in this area is desirable, not essential Operate within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/ More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

Databricks Tech Lead – Azure Data Engineering (Contract) We are looking for a highly technical, hands-on Databricks Tech Lead to join a large-scale digital transformation programme within the Energy Trading space. This role will focus on building, optimising, and scaling a data lake on Azure with particular emphasis on streaming pipelines, Delta Live Tables (DLT … and cost-efficient architecture. Role: Designing and developing streaming and batch data pipelines in Databricks (including DLT). Managing structured streaming in continuous mode, ensuring low-latency data delivery. Collaborating with architects, consultants, and business stakeholders to build a unified Azure-based lakehouse. Troubleshooting and optimising performance bottlenecks in high-volume streaming systems. Applying best practices … for cloud cost management and efficient Databricks resource usage. Integrating data from on-prem and cloud sources using tools like Qlik Replicate (CDC). Supporting future phases involving data science and machine learning enablement. Tech Environment Platform: Azure Core Tools: Databricks, Delta Live Tables, Qlik Replicate (CDC) Architecture: Hybrid (on-prem + cloud) Data More ❯

Databricks Tech Lead – Azure Data Engineering (Contract) We are looking for a highly technical, hands-on Databricks Tech Lead to join a large-scale digital transformation programme within the Energy Trading space. This role will focus on building, optimising, and scaling a data lake on Azure with particular emphasis on streaming pipelines, Delta Live Tables (DLT … and cost-efficient architecture. Role: Designing and developing streaming and batch data pipelines in Databricks (including DLT). Managing structured streaming in continuous mode, ensuring low-latency data delivery. Collaborating with architects, consultants, and business stakeholders to build a unified Azure-based lakehouse. Troubleshooting and optimising performance bottlenecks in high-volume streaming systems. Applying best practices … for cloud cost management and efficient Databricks resource usage. Integrating data from on-prem and cloud sources using tools like Qlik Replicate (CDC). Supporting future phases involving data science and machine learning enablement. Tech Environment Platform: Azure Core Tools: Databricks, Delta Live Tables, Qlik Replicate (CDC) Architecture: Hybrid (on-prem + cloud) Data More ❯

About ClearWatt ClearWatt is on a mission to make electric vehicle ownership transparent and accessible. Our platform combines real-world data, telematics, and proprietary analytics to deliver trusted insights into EV battery health, real-world range, and performance. We partner with car dealers, leasing companies, energy companies and consumers to help them understand the true condition of electric … vehicles - unlocking confidence across the EV ecosystem. We’re scaling rapidly, building the UK’s most comprehensive EV data intelligence platform - spanning OBD diagnostics, API-based vehicle integrations, and real-time energy data streams. The Role We’re looking for a skilled Backend Developer with strong experience in C# and the .NET ecosystem to help us … scale and evolve our core data and API infrastructure. You’ll play a central role in designing, building, and optimising services that power our EV testing, reporting, and data-sharing products. This role is primarily backend-focused, but familiarity with modern frontend frameworks (React/React Native/Next.js) is a bonus. Key Responsibilities Develop and More ❯

About ClearWatt ClearWatt is on a mission to make electric vehicle ownership transparent and accessible. Our platform combines real-world data, telematics, and proprietary analytics to deliver trusted insights into EV battery health, real-world range, and performance. We partner with car dealers, leasing companies, energy companies and consumers to help them understand the true condition of electric … vehicles - unlocking confidence across the EV ecosystem. We’re scaling rapidly, building the UK’s most comprehensive EV data intelligence platform - spanning OBD diagnostics, API-based vehicle integrations, and real-time energy data streams. The Role We’re looking for a skilled Backend Developer with strong experience in C# and the .NET ecosystem to help us … scale and evolve our core data and API infrastructure. You’ll play a central role in designing, building, and optimising services that power our EV testing, reporting, and data-sharing products. This role is primarily backend-focused, but familiarity with modern frontend frameworks (React/React Native/Next.js) is a bonus. Key Responsibilities Develop and More ❯

working with a global Oil, Gas and Energy firm in London seeking an experienced Contract Python Developer for a high-impact engagement. You’ll be contributing to scalable, data-driven platforms and back-end systems in a complex and fast-moving environment, collaborating across teams to deliver cutting-edge solutions in the energy sector. Responsibilities of the Contract … Python Developer (Commodities) Lead the design, development and optimisation of robust Python-based applications and services Build scalable, secure APIs and data pipelines using modern frameworks and tools Work closely with DevOps and data teams to deploy and maintain microservices in cloud environments (e.g., AWS, Azure) Write clean, maintainable code with strong testing coverage using tools … such as Django, Flask, or FastAPI Familiarity with containerisation (Docker), CI/CD pipelines, and infrastructure-as-code tools Exposure to cloud platforms (preferably AWS or Azure) and data engineering workflows Proficiency in relational and non-relational databases (e.g. PostgreSQL, MongoDB) Experience working in Agile environments and version control systems like Git Prior experience in the Oil, Gas More ❯

working with a global Oil, Gas and Energy firm in London seeking an experienced Contract Python Developer for a high-impact engagement. You’ll be contributing to scalable, data-driven platforms and back-end systems in a complex and fast-moving environment, collaborating across teams to deliver cutting-edge solutions in the energy sector. Responsibilities of the Contract … Python Developer (Commodities) Lead the design, development and optimisation of robust Python-based applications and services Build scalable, secure APIs and data pipelines using modern frameworks and tools Work closely with DevOps and data teams to deploy and maintain microservices in cloud environments (e.g., AWS, Azure) Write clean, maintainable code with strong testing coverage using tools … such as Django, Flask, or FastAPI Familiarity with containerisation (Docker), CI/CD pipelines, and infrastructure-as-code tools Exposure to cloud platforms (preferably AWS or Azure) and data engineering workflows Proficiency in relational and non-relational databases (e.g. PostgreSQL, MongoDB) Experience working in Agile environments and version control systems like Git Prior experience in the Oil, Gas More ❯

Goal We’re looking for an AI Platform Specialist to build and optimize the infrastructure powering our AI ambitions. In this hands-on role, you’ll design scalable data pipelines, manage cloud services, and implement infrastructure-as-code to enable secure, efficient, and high-performance AI workflows. Your work will ensure data is not just available … but ready for AI at scale. You’ll collaborate closely with ML engineers, software developers, and stakeholders to unlock siloed data, improve compliance, and drive platform reliability. This is your chance to shape the technical foundation behind real-world AI impact in a high-growth, high-regulation industry. The Player You bring deep expertise in cloud infrastructure (AWS … Azure, or GCP), data engineering, CI/CD, and IaC tools like Terraform. You thrive in cross-functional environments, balance performance with governance, and know how to turn complex systems into seamless pipelines. If you're ready to empower enterprise AI with next-gen infrastructure — we want to hear from you. What we can offer YOU Why would More ❯

do This is a senior, strategic consulting role — driving AI/ML security architecture, influencing C-suite direction, and ensuring security-by-design across complex AI systems and data pipelines. You’ll define and deliver AI security strategies, develop reference architectures, and provide expert guidance on risk management, data protection, and ethical AI governance. You’ll … advisory. Deep understanding of AI/ML security architecture, adversarial ML mitigation, and model governance. Current security certifications such as CISSP or CISM. Proven experience across cloud security, data privacy, and DevSecOps. Familiarity with frameworks including NIST, ISO 27001, TOGAF, SABSA. Excellent communication and stakeholder influence skills. Eligible for UK Security Clearance (SC). If you’re an More ❯

do This is a senior, strategic consulting role — driving AI/ML security architecture, influencing C-suite direction, and ensuring security-by-design across complex AI systems and data pipelines. You’ll define and deliver AI security strategies, develop reference architectures, and provide expert guidance on risk management, data protection, and ethical AI governance. You’ll … advisory. Deep understanding of AI/ML security architecture, adversarial ML mitigation, and model governance. Current security certifications such as CISSP or CISM. Proven experience across cloud security, data privacy, and DevSecOps. Familiarity with frameworks including NIST, ISO 27001, TOGAF, SABSA. Excellent communication and stakeholder influence skills. Eligible for UK Security Clearance (SC). If you’re an More ❯

scalable and interconnected platform services. These services range from engagement tools such as polling and prediction to services such as push notification systems and high throughput analytics and data pipelines. The successful candidate will have the opportunity to work on a wide variety of projects across the sports sector. They will also be able to assist with maintaining … way from the requirements analysis stage to the delivery of the feature. Maintaining and improving all existing backend Symfony services. Maintaining and improving our first party analytics and data pipelines and ensuring integration of new features with these pipelines, for example tracking poll votes and failed push notification sends. Working with specialist Drupal, Magento, Frontend, iOS and Android … teams on API contracts to connect platforms, for example poll choice management APIs, sports data APIs, ticketing APIs. Participating in team code review practices, reviewing code from other teams and working with them to review your own code. Leading end-to-end testing to ensure high quality delivery to the product and project teams, involving automated unit and More ❯

scalable and interconnected platform services. These services range from engagement tools such as polling and prediction to services such as push notification systems and high throughput analytics and data pipelines. The successful candidate will have the opportunity to work on a wide variety of projects across the sports sector. They will also be able to assist with maintaining … way from the requirements analysis stage to the delivery of the feature. Maintaining and improving all existing backend Symfony services. Maintaining and improving our first party analytics and data pipelines and ensuring integration of new features with these pipelines, for example tracking poll votes and failed push notification sends. Working with specialist Drupal, Magento, Frontend, iOS and Android … teams on API contracts to connect platforms, for example poll choice management APIs, sports data APIs, ticketing APIs. Participating in team code review practices, reviewing code from other teams and working with them to review your own code. Leading end-to-end testing to ensure high quality delivery to the product and project teams, involving automated unit and More ❯

ensuring stability, consistency, and minimal disruption Quickly learn and manage delivery from our legacy CEP (Blueshift), ensuring smooth transition off the platform and into Braze. Partner with CRM, Data, and Engineering teams to configure Braze instances, validate SDK integrations, and manage data schema alignment. Own the migration playbook, ensuring documentation and learnings are captured and shared … across markets. Conduct UAT and QA for migrated data and campaigns, ensuring compliance, deliverability, and performance tracking. Optimisation & Capability Growth (Post-Migration) Lead the exploration, testing, and rollout of new Braze features (e.g. Content Cards, In-App Messaging, Personalised Paths, AI Catalyst). Partner with CRM stakeholders on the implementation of our new global contact strategy leveraging Braze … HTML, CSS, Liquid) to ensure communications are scalable, maintainable, and consistently high quality — keeping our customer comms visually outstanding and the envy of competitors. Champion best practice in data-driven targeting, segmentation, and personalisation. Ensure documentation and knowledge sharing so CRM teams across markets can self-serve effectively. Ongoing Responsibilities Act as the primary subject matter expert for More ❯

ensuring stability, consistency, and minimal disruption Quickly learn and manage delivery from our legacy CEP (Blueshift), ensuring smooth transition off the platform and into Braze. Partner with CRM, Data, and Engineering teams to configure Braze instances, validate SDK integrations, and manage data schema alignment. Own the migration playbook, ensuring documentation and learnings are captured and shared … across markets. Conduct UAT and QA for migrated data and campaigns, ensuring compliance, deliverability, and performance tracking. Optimisation & Capability Growth (Post-Migration) Lead the exploration, testing, and rollout of new Braze features (e.g. Content Cards, In-App Messaging, Personalised Paths, AI Catalyst). Partner with CRM stakeholders on the implementation of our new global contact strategy leveraging Braze … HTML, CSS, Liquid) to ensure communications are scalable, maintainable, and consistently high quality — keeping our customer comms visually outstanding and the envy of competitors. Champion best practice in data-driven targeting, segmentation, and personalisation. Ensure documentation and knowledge sharing so CRM teams across markets can self-serve effectively. Ongoing Responsibilities Act as the primary subject matter expert for More ❯

Mid-Level Engineer - Python, SQL, Airflow, BigQuery, DBT, GCP I've partnered with a high growth Product led tech company based in Central London, who are expanding their Data function, and are looking for a highly skilled, Mid-Level Data Engineer to join their team. In this role, you'll be reporting directly to the Head … of Engineering, where the Infrastructure and Data Platform will be the focus, building robust and highly scalable Data Pipelines, whilst working with stakeholders across the business, to help make data driven decisions. Mid-Level Engineer - Python, SQL, Airflow, BigQuery, DBT, GCP Key skills and experience required: 4+ years of experience in Data Engineering Python & SQL GCP BigQuery Airflow DBT Familiarity with Data Ingestion (ideally in a SaaS context) Product mindset This role is a hybrid role with 3-days a week required in Central London offices Base salary of £60k-£80k depending on skills and experience Bonus + excellent benefits package included Apply for more information! Mid-Level More ❯

Software Engineer- Python- Data Science and Machine Learning Team The RJC Group is looking for an experienced Software Engineer to join a growing Data Science & Machine Learning team within a global energy and commodities company. This is a hands-on role building the tools, pipelines, and infrastructure that power real-world ML applications across commercial and … operational areas. What You’ll Do Work with data scientists to deliver scalable ML solutions Build and maintain clean, reliable data pipelines and internal tools Collaborate with technical and non-technical stakeholders What You Bring 3–5+ years of experience in software engineering (Python) Experience with fastAPI, cloud platforms (AWS, Azure, or GCP), Docker. Bonus: experience More ❯

across a diverse range of global strategies. Responsibilities: Design and build microservices for real-time calculation of VaR, stress tests, scenarios, and intraday limit utilization. Develop high-throughput data pipelines for market, trade, and reference data, with a focus on achieving sub-second latency. Create intuitive, web-based dashboards for portfolio managers and risk officers to … years of experience building production systems in a front-office trading, hedge fund, or high-growth tech environment. Python skills for numerical computing, along with strong data engineering fundamentals. Experience with C++ or Rust is welcomed. Familiarity with risk concepts such as VaR, sensitivities, and margin, as well as an understanding of market microstructure. Proven ability to make More ❯

stores, and ops teams to see your code in action. 🧱 What You’ll Be Building Scalable backend services and operational tooling using Node.js, TypeScript, and AWS APIs and data pipelines to support logistics, supply chain, and real-time store operations Event-driven systems and distributed architecture for fast, fault-tolerant execution Back-office tools and dashboards for internal … teams—integrated directly with on-the-ground workflows Systems to optimise KPIs like order accuracy, worker productivity, and inventory speed 🔧 Tech Stack Backend: Node.js, TypeScript, Express Data: PostgreSQL, MongoDB, event streams Infra: AWS (Lambda, DynamoDB, EventBridge), GCP, Docker, CI/CD pipelines Architecture: Serverless, microservices, event-driven Tooling: Monitoring, dashboarding, dev tooling, cloud infra automation ✅ What You Bring … 5+ years backend experience with Node.js and TypeScript Deep experience with SQL/NoSQL databases and data modelling Strong understanding of cloud infrastructure (AWS or GCP) and DevOps Experience building and maintaining scalable APIs and backend systems Ability to lead architecture, define standards, and raise the bar for engineering quality Great collaborator with product, design, and ops teams More ❯

stores, and ops teams to see your code in action. 🧱 What You’ll Be Building Scalable backend services and operational tooling using Node.js, TypeScript, and AWS APIs and data pipelines to support logistics, supply chain, and real-time store operations Event-driven systems and distributed architecture for fast, fault-tolerant execution Back-office tools and dashboards for internal … teams—integrated directly with on-the-ground workflows Systems to optimise KPIs like order accuracy, worker productivity, and inventory speed 🔧 Tech Stack Backend: Node.js, TypeScript, Express Data: PostgreSQL, MongoDB, event streams Infra: AWS (Lambda, DynamoDB, EventBridge), GCP, Docker, CI/CD pipelines Architecture: Serverless, microservices, event-driven Tooling: Monitoring, dashboarding, dev tooling, cloud infra automation ✅ What You Bring … 5+ years backend experience with Node.js and TypeScript Deep experience with SQL/NoSQL databases and data modelling Strong understanding of cloud infrastructure (AWS or GCP) and DevOps Experience building and maintaining scalable APIs and backend systems Ability to lead architecture, define standards, and raise the bar for engineering quality Great collaborator with product, design, and ops teams More ❯

the potential (or ambition) to lead technology at a company-defining level. 🔍 What You'll Do Build and own core infrastructure and backend systems Design and implement APIs, data models, and scalable services Collaborate closely with founders on product direction and technical architecture Ship code fast, validate assumptions, and iterate Lay the groundwork for the future engineering team … doesn’t scare you; in fact, you prefer it Experienced with cloud infrastructure — AWS, GCP, or similar; IaC (Terraform, Pulumi, etc.) is a plus Competent with databases and data modeling — SQL and NoSQL, designing schema for scale Comfortable writing tests and deploying production systems — you care about code quality and reliability ⭐ Bonus Experience (Not Required, but Valuable) Worked … enterprise customers Experience with applied machine learning or MLOps (not just academic or notebook-based work) Exposure to event-driven architectures, streaming systems (Kafka, etc.), or real-time data pipelines Frontend experience in React, TypeScript, or similar modern frameworks 📈 What We Offer Early-stage equity and the opportunity to shape foundational decisions High ownership, zero bureaucracy — build what More ❯

the potential (or ambition) to lead technology at a company-defining level. 🔍 What You'll Do Build and own core infrastructure and backend systems Design and implement APIs, data models, and scalable services Collaborate closely with founders on product direction and technical architecture Ship code fast, validate assumptions, and iterate Lay the groundwork for the future engineering team … doesn’t scare you; in fact, you prefer it Experienced with cloud infrastructure — AWS, GCP, or similar; IaC (Terraform, Pulumi, etc.) is a plus Competent with databases and data modeling — SQL and NoSQL, designing schema for scale Comfortable writing tests and deploying production systems — you care about code quality and reliability ⭐ Bonus Experience (Not Required, but Valuable) Worked … enterprise customers Experience with applied machine learning or MLOps (not just academic or notebook-based work) Exposure to event-driven architectures, streaming systems (Kafka, etc.), or real-time data pipelines Frontend experience in React, TypeScript, or similar modern frameworks 📈 What We Offer Early-stage equity and the opportunity to shape foundational decisions High ownership, zero bureaucracy — build what More ❯

quo, building products people love, and having fun along the way. We’re looking for a talented Python Engineer who’s passionate about creating an exceptional real-time data platform that will process and deliver data instantaneously; allowing for instant insights and actions. In finance, these systems are crucial because markets move very quickly, and delays … of even milliseconds can have big consequences. Essential skills: 3+ years of experience in Python development. 3+ with open-source real-time data feeds (Amazon Kinesis, Apache Kafka, Apache Pulsar or Redpanda) Exposure building and managing data pipelines in production. Experience integrating serverless functions (AWS, Azure or GCP). Passion for fintech and building products that … make a real difference. Day to day: Develop and maintain our real-time data streaming pipelines. Write clean, reusable and scalable code. Optimise performance and troubleshoot issues across the platform. Stay up to date with the latest development trends and tools. Collaborate with product and engineering to deliver high-quality features. Benefits: Opportunity to shape a product used More ❯

quo, building products people love, and having fun along the way. We’re looking for a talented Python Engineer who’s passionate about creating an exceptional real-time data platform that will process and deliver data instantaneously; allowing for instant insights and actions. In finance, these systems are crucial because markets move very quickly, and delays … of even milliseconds can have big consequences. Essential skills: 3+ years of experience in Python development. 3+ with open-source real-time data feeds (Amazon Kinesis, Apache Kafka, Apache Pulsar or Redpanda) Exposure building and managing data pipelines in production. Experience integrating serverless functions (AWS, Azure or GCP). Passion for fintech and building products that … make a real difference. Day to day: Develop and maintain our real-time data streaming pipelines. Write clean, reusable and scalable code. Optimise performance and troubleshoot issues across the platform. Stay up to date with the latest development trends and tools. Collaborate with product and engineering to deliver high-quality features. Benefits: Opportunity to shape a product used More ❯