126 to 150 of 161 Data Pipeline Jobs in the City of London

Senior Software Engineer - Backend & Data | Fully Remote (UK) Founding Engineer | Python | GCP | Terraform | Event-Based Systems | Data Pipelines An innovative, mission-driven LegalTech SaaS start-up is looking for a Senior Product Engineer (Backend & Data) with deep experience in Python, Product development, and Data engineering. Role Information: Salary : Up to … over technical decisions from day one. You’ll take technical ownership of the backend to ensure it’s robust, scalable, and ready for real customers, while adding new Data-driven features, optimising performance, and shaping the long-term Product roadmap. While your focus will be backend systems in Python, you’ll also work across the stack, collaborate directly … a strong Product mindset to every decision. This role is ideal for someone who thrives on turning early-stage ideas into scalable, market-ready Products backed by solid Data pipelines. Senior Product Engineer - Highlights: Enhance and scale backend services in Python Architect event-driven solutions in GCP with Pub/Sub & Cloud Run Build and optimise Python code More ❯

multiple teams and contribute on key decisions. Provide solutions to problems for their immediate team and across multiple teams. Collaborate with cross-functional teams to develop and deploy data-driven insights. As an Infrastructure Automation Engineer you will be: Required to work independently and be self-motivated, leading a team of other Automation Infrastructure Engineers and Analysts Experienced … in the following: Infrastructure Development and Automation: Building use cases and creating infrastructure applications using programming languages such as C#, C++, and Python to process data locally Containerization and Orchestration: Using tools like Docker and Kubernetes to package and manage applications for scalable and efficient deployment. Packaged Application and industry standard APIs such as Revitt, MEF and TMFORUM. … and Resource Management: Managing resources efficiently via deployment of scalable infrastructure. Includes knowledge of containerization, virtualization, and orchestration tools to dynamically allocate resources based on demand. Develop innovative data solutions to optimize data generation, collection, and processing. Collaborate with cross-functional teams to ensure data quality and integrity. We are looking for experience in More ❯

multiple teams and contribute on key decisions. Provide solutions to problems for their immediate team and across multiple teams. Collaborate with cross-functional teams to develop and deploy data-driven insights. As an Infrastructure Automation Engineer you will be: Required to work independently and be self-motivated, leading a team of other Automation Infrastructure Engineers and Analysts Experienced … in the following: Infrastructure Development and Automation: Building use cases and creating infrastructure applications using programming languages such as C#, C++, and Python to process data locally Containerization and Orchestration: Using tools like Docker and Kubernetes to package and manage applications for scalable and efficient deployment. Packaged Application and industry standard APIs such as Revitt, MEF and TMFORUM. … and Resource Management: Managing resources efficiently via deployment of scalable infrastructure. Includes knowledge of containerization, virtualization, and orchestration tools to dynamically allocate resources based on demand. Develop innovative data solutions to optimize data generation, collection, and processing. Collaborate with cross-functional teams to ensure data quality and integrity. We are looking for experience in More ❯

value of technology and build a more sustainable, more inclusive world. Your Role: We are seeking a highly experienced and innovative Microsoft Fabric Architect to lead enterprise-scale data modernization and AI initiatives. This role requires deep technical expertise in Microsoft Fabric and related services, with the ability to translate business needs into scalable architecture on the Microsoft … platform. The ideal candidate will be a thought leader, trusted advisor, and hands-on architect capable of shaping next-generation data platforms and driving adoption across stakeholder groups. Lead end-to-end architecture design and implementation for Microsoft Fabric across enterprise data programs. Collaborate with business and technical stakeholders to align Fabric capabilities with strategic goals. … Architect unified data pipelines using Fabric Data Factory, Power BI, and Lakehouse components. Drive technical delivery, ensuring best practices in performance, security, and cost optimization. Conduct enablement sessions, workshops, and architecture reviews to support development teams. Develop reusable frameworks, accelerators, and governance models for scalable deployments. Evangelize new Fabric capabilities and represent the organization in partner More ❯

value of technology and build a more sustainable, more inclusive world. Your Role: We are seeking a highly experienced and innovative Microsoft Fabric Architect to lead enterprise-scale data modernization and AI initiatives. This role requires deep technical expertise in Microsoft Fabric and related services, with the ability to translate business needs into scalable architecture on the Microsoft … platform. The ideal candidate will be a thought leader, trusted advisor, and hands-on architect capable of shaping next-generation data platforms and driving adoption across stakeholder groups. Lead end-to-end architecture design and implementation for Microsoft Fabric across enterprise data programs. Collaborate with business and technical stakeholders to align Fabric capabilities with strategic goals. … Architect unified data pipelines using Fabric Data Factory, Power BI, and Lakehouse components. Drive technical delivery, ensuring best practices in performance, security, and cost optimization. Conduct enablement sessions, workshops, and architecture reviews to support development teams. Develop reusable frameworks, accelerators, and governance models for scalable deployments. Evangelize new Fabric capabilities and represent the organization in partner More ❯

this role, you will be responsible for: Delivering strategic advisory, engineering delivery, and enablement services to Government and commercial clients. Solve user problems by drawing source and captured data into a single foundational data model and building operational workflows that materially change the decisions made Work closely with clients, to tailor products and services Design solutions … for client problem sets based on their descriptions and by exploration of the data landscape Assist in the design, development, and maintenance of data pipelines and ETL processes to build data and action models to address the workflow needs Build and edit operational workflows, including front ends and decision-support toolsets, inclusive of native … company with strong ideals and ambitions. You will work closely with teams of experienced and early talent engineers on a variety of projects, gaining hands-on experience in data management, analysis, and AI technologies, on profound and important problems. This represents an excellent opportunity to secure a permanent role with a growing and dynamic organisation. For further information More ❯

value encompassing content creation, predictive analytics and process automation; select and integrate optimal AI technologies, conduct training workshops for employees at all levels and partner with software and data engineering teams to ensure the infrastructure and data pipelines are in place to support the needs of Generative AI applications. This is an impactful role with a … s a friendly and collaborative environment with casual dress code and a range of facilities. About you: You are degree educated in a relevant discipline i.e. Computer Science, Data Science or closely related, likely to have MSc in Artificial Intelligence or Machine Learning You're tech savvy with experience of integrating emerging technologies, with your most recent experience More ❯

Position: Data Engineer Location: Central London or Bristol (3 days per week onsite) Salary: £600 to £650 p.d inside IR35 Security Clearance: SC Clearance Required About the Role A leading IT consultancy delivering innovative solutions to the UK public sector is seeking an experienced Data Engineer to support a high-profile programme. You'll collaborate with … cross-functional teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation … Docker/Kubernetes) - experience in this area is desirable, not essential Operate within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/ More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

makers in a live-market environment. Responsibilities: Architect and code resilient applications for trading, liquidity management, and funding operations using Python as the core language, with integration into data platforms and tools. Engage directly with traders and treasury stakeholders during live situations, adapting to evolving requirements and delivering solutions on the fly. Design and optimise data pipelines for real-time and historical treasury data, ensuring accuracy and performance in analytics. Work with operations teams to deploy, monitor, and scale systems, ensuring uptime and quick incident response. Identify technology gaps and inefficiencies, experimenting with new tools and design patterns to enhance speed and quality. Experience: 5-15 years of software engineering experience, ideally … in Treasury or from a hedge fund/buy-side firm Expertise in Python for designing and optimising complex applications Strong proficiency in data tools like Pandas, NumPy, SQL, and real-time data processing Familiarity with DevOps practices, including Linux, cloud platforms, and CI/CD (Docker/Kubernetes is desirable but not essential). Strong More ❯

About ClearWatt ClearWatt is on a mission to make electric vehicle ownership transparent and accessible. Our platform combines real-world data, telematics, and proprietary analytics to deliver trusted insights into EV battery health, real-world range, and performance. We partner with car dealers, leasing companies, energy companies and consumers to help them understand the true condition of electric … vehicles - unlocking confidence across the EV ecosystem. We’re scaling rapidly, building the UK’s most comprehensive EV data intelligence platform - spanning OBD diagnostics, API-based vehicle integrations, and real-time energy data streams. The Role We’re looking for a skilled Backend Developer with strong experience in C# and the .NET ecosystem to help us … scale and evolve our core data and API infrastructure. You’ll play a central role in designing, building, and optimising services that power our EV testing, reporting, and data-sharing products. This role is primarily backend-focused, but familiarity with modern frontend frameworks (React/React Native/Next.js) is a bonus. Key Responsibilities Develop and More ❯

About ClearWatt ClearWatt is on a mission to make electric vehicle ownership transparent and accessible. Our platform combines real-world data, telematics, and proprietary analytics to deliver trusted insights into EV battery health, real-world range, and performance. We partner with car dealers, leasing companies, energy companies and consumers to help them understand the true condition of electric … vehicles - unlocking confidence across the EV ecosystem. We’re scaling rapidly, building the UK’s most comprehensive EV data intelligence platform - spanning OBD diagnostics, API-based vehicle integrations, and real-time energy data streams. The Role We’re looking for a skilled Backend Developer with strong experience in C# and the .NET ecosystem to help us … scale and evolve our core data and API infrastructure. You’ll play a central role in designing, building, and optimising services that power our EV testing, reporting, and data-sharing products. This role is primarily backend-focused, but familiarity with modern frontend frameworks (React/React Native/Next.js) is a bonus. Key Responsibilities Develop and More ❯

Goal We’re looking for an AI Platform Specialist to build and optimize the infrastructure powering our AI ambitions. In this hands-on role, you’ll design scalable data pipelines, manage cloud services, and implement infrastructure-as-code to enable secure, efficient, and high-performance AI workflows. Your work will ensure data is not just available … but ready for AI at scale. You’ll collaborate closely with ML engineers, software developers, and stakeholders to unlock siloed data, improve compliance, and drive platform reliability. This is your chance to shape the technical foundation behind real-world AI impact in a high-growth, high-regulation industry. The Player You bring deep expertise in cloud infrastructure (AWS … Azure, or GCP), data engineering, CI/CD, and IaC tools like Terraform. You thrive in cross-functional environments, balance performance with governance, and know how to turn complex systems into seamless pipelines. If you're ready to empower enterprise AI with next-gen infrastructure — we want to hear from you. What we can offer YOU Why would More ❯

do This is a senior, strategic consulting role — driving AI/ML security architecture, influencing C-suite direction, and ensuring security-by-design across complex AI systems and data pipelines. You’ll define and deliver AI security strategies, develop reference architectures, and provide expert guidance on risk management, data protection, and ethical AI governance. You’ll … advisory. Deep understanding of AI/ML security architecture, adversarial ML mitigation, and model governance. Current security certifications such as CISSP or CISM. Proven experience across cloud security, data privacy, and DevSecOps. Familiarity with frameworks including NIST, ISO 27001, TOGAF, SABSA. Excellent communication and stakeholder influence skills. Eligible for UK Security Clearance (SC). If you’re an More ❯

do This is a senior, strategic consulting role — driving AI/ML security architecture, influencing C-suite direction, and ensuring security-by-design across complex AI systems and data pipelines. You’ll define and deliver AI security strategies, develop reference architectures, and provide expert guidance on risk management, data protection, and ethical AI governance. You’ll … advisory. Deep understanding of AI/ML security architecture, adversarial ML mitigation, and model governance. Current security certifications such as CISSP or CISM. Proven experience across cloud security, data privacy, and DevSecOps. Familiarity with frameworks including NIST, ISO 27001, TOGAF, SABSA. Excellent communication and stakeholder influence skills. Eligible for UK Security Clearance (SC). If you’re an More ❯

quo, building products people love, and having fun along the way. We’re looking for a talented Python Engineer who’s passionate about creating an exceptional real-time data platform that will process and deliver data instantaneously; allowing for instant insights and actions. In finance, these systems are crucial because markets move very quickly, and delays … of even milliseconds can have big consequences. Essential skills: 3+ years of experience in Python development. 3+ with open-source real-time data feeds (Amazon Kinesis, Apache Kafka, Apache Pulsar or Redpanda) Exposure building and managing data pipelines in production. Experience integrating serverless functions (AWS, Azure or GCP). Passion for fintech and building products that … make a real difference. Day to day: Develop and maintain our real-time data streaming pipelines. Write clean, reusable and scalable code. Optimise performance and troubleshoot issues across the platform. Stay up to date with the latest development trends and tools. Collaborate with product and engineering to deliver high-quality features. Benefits: Opportunity to shape a product used More ❯

quo, building products people love, and having fun along the way. We’re looking for a talented Python Engineer who’s passionate about creating an exceptional real-time data platform that will process and deliver data instantaneously; allowing for instant insights and actions. In finance, these systems are crucial because markets move very quickly, and delays … of even milliseconds can have big consequences. Essential skills: 3+ years of experience in Python development. 3+ with open-source real-time data feeds (Amazon Kinesis, Apache Kafka, Apache Pulsar or Redpanda) Exposure building and managing data pipelines in production. Experience integrating serverless functions (AWS, Azure or GCP). Passion for fintech and building products that … make a real difference. Day to day: Develop and maintain our real-time data streaming pipelines. Write clean, reusable and scalable code. Optimise performance and troubleshoot issues across the platform. Stay up to date with the latest development trends and tools. Collaborate with product and engineering to deliver high-quality features. Benefits: Opportunity to shape a product used More ❯

We are partnering with a rapidly scaling Palantir implementation partner, expanding across all seniority levels, having secured a strong pipeline of projects across various industries. This is an exciting opportunity to specialise in one of the fastest-growing technology markets and collaborate with some of the most experienced leaders in the field. Responsibilities: Develop and maintain data More ❯

We are partnering with a rapidly scaling Palantir implementation partner, expanding across all seniority levels, having secured a strong pipeline of projects across various industries. This is an exciting opportunity to specialise in one of the fastest-growing technology markets and collaborate with some of the most experienced leaders in the field. Responsibilities: Develop and maintain data More ❯

all stages of the AI lifecycle. Act as the technical SME, providing guidance and solutions to complex security challenges in AI/ML deployments. Work closely with engineering, data science, and DevSecOps teams to embed security best practices. Assess and mitigate risks specific to AI models, data pipelines, and ML/AI frameworks. Influence security strategy … edge AI security initiatives. Experience & Skills Required: Proven hands-on experience securing ML/AI systems (not theoretical knowledge). Deep technical expertise in AI/ML architectures, data pipelines, and model lifecycle security. Strong understanding of threat modeling, risk assessment, and security controls in AI/ML contexts. Previous experience as a Security Architect or senior security More ❯

all stages of the AI lifecycle. Act as the technical SME, providing guidance and solutions to complex security challenges in AI/ML deployments. Work closely with engineering, data science, and DevSecOps teams to embed security best practices. Assess and mitigate risks specific to AI models, data pipelines, and ML/AI frameworks. Influence security strategy … edge AI security initiatives. Experience & Skills Required: Proven hands-on experience securing ML/AI systems (not theoretical knowledge). Deep technical expertise in AI/ML architectures, data pipelines, and model lifecycle security. Strong understanding of threat modeling, risk assessment, and security controls in AI/ML contexts. Previous experience as a Security Architect or senior security More ❯

We believe in better . And we make it happen . Better content . Better products. And better careers . Working in Tech, Product or Data at Sky is about building the next and the new . From broadband to broadcast, streaming to mobile, SkyQ to Sky Glass, we never stand still . We optimise and innovate . … around observability , incident response , and reducing operational toil . Proficiency in scripting and programming languages such as C# and Python . Experience working with APIs , telemetry systems , and data pipelines for metrics and logs . Ability to design and maintain custom dashboards , alerting logic , and monitoring workflows . Knowledge of containerized environments ( e.g ., Docker, Kubernetes ) and cloud … washed , and even get pampered at our beauty salon. We'd love to hear from you Inventive , forward-thinking minds come together to work in Tech, Product and Data at Sky . It's a place where you can explore what if , how far, and what next . But better doesn't stop at what we do, it More ❯

Equity Potential Really excited to be working with this high-growth InsurTech who are transforming how risk is traded in the commercial insurance market using bespoke forward thinking data methods. They’re building a smarter, data-driven exchange that connects brokers and insurers, and are now looking for a Senior Exposure Management Analyst to lead catastrophe … carriers to embed exposure insights into algorithmic underwriting and shape market best practices. Key Responsibilities: Design and implement exposure and accumulation frameworks Lead the development of model-ready data pipelines and reporting tools Collaborate with model vendors and internal teams to expand capabilities Guide pre- and post-cat event analysis and dynamic rule setting Act as the subject … in property (re)insurance (This is a must have requirement) Strong SQL and database skills Familiarity with GIS tools (QGIS, ArcGIS); Python preferred Proven ability to communicate complex data to technical and non-technical audiences In addition to a competitive salary (variable depending on experience level), the package includes a performance-based bonus and potential equity options. If More ❯

Equity Potential Really excited to be working with this high-growth InsurTech who are transforming how risk is traded in the commercial insurance market using bespoke forward thinking data methods. They’re building a smarter, data-driven exchange that connects brokers and insurers, and are now looking for a Senior Exposure Management Analyst to lead catastrophe … carriers to embed exposure insights into algorithmic underwriting and shape market best practices. Key Responsibilities: Design and implement exposure and accumulation frameworks Lead the development of model-ready data pipelines and reporting tools Collaborate with model vendors and internal teams to expand capabilities Guide pre- and post-cat event analysis and dynamic rule setting Act as the subject … in property (re)insurance (This is a must have requirement) Strong SQL and database skills Familiarity with GIS tools (QGIS, ArcGIS); Python preferred Proven ability to communicate complex data to technical and non-technical audiences In addition to a competitive salary (variable depending on experience level), the package includes a performance-based bonus and potential equity options. If More ❯

both new and existing secure transformation projects. You’ll play a key role in architecting and maintaining scalable, security-first cloud infrastructure using AWS, Kubernetes, and real-time data streaming pipelines. This is a hands-on role for engineers who thrive in high-assurance environments and enjoy driving automation and DevSecOps best practices. What’s on Offer Highly … impact on secure solutions shaping UK defence programmes What You’ll Be Doing Designing and supporting secure AWS multi-account environments Driving automation, container orchestration, and real-time data pipelines Leading initiatives in infrastructure-as-code, CI/CD, and platform monitoring Enabling robust security practices across DevOps delivery teams Core Skills & Experience Strong experience with AWS MSK More ❯