26 to 50 of 59 Data Pipeline Jobs in the City of London

British Airways is seeking a Principal Data Engineer to lead data engineering practices across delivery teams. You will design, build, test, and deploy secure and scalable data pipelines, ensuring availability and performance of data products. You will lead agile ceremonies and coach … engineers while on-call to support live data products. You will also manage senior data engineers, set standards for data processing documentation, and collaborate with stakeholders to drive reliable, scalable data #J-18808-Ljbffr ...

Lead Data Engineer (Permanent, London on-site) A high-growth AI software company is looking for a Lead Data Engineer to own and scale the data foundations behind an AI-driven operations platform used by property teams. This is a hands-on, high-ownership … role where you'll define the data architecture, set engineering standards and build systems that power product analytics and modern AI capabilities. What you'll do Architect and build scalable data pipelines and distributed data systems Design ingestion, transformation and storage patterns for operational ...

Funds (ETF) sector, is seeking an experienced Lead Python Developer to play a key role in shaping the future of their technology and data platforms. This is a rare opportunity to join a highly respected organisation operating at the intersection of technology, data and financial markets. … delivery of business-critical solutions while working closely with senior stakeholders across the organisation. Key Responsibilities: Leading the design and development of scalable data platforms and applications Building and optimising Python-based solutions and automation tools Developing and maintaining PostgreSQL databases and data models Delivering Azure ...

Senior Data Engineer | London (Camden) | Hybrid (4 days in office) We're partnering with a fast-growing AI technology business that's transforming how businesses make operational decisions through data and machine learning. They're looking for a Senior Data Engineer to take ownership … modern, AWS-native data platform, building scalable pipelines and production-grade ML infrastructure that powers real-time analytics and intelligent decision-making. Within 3/6 months, you will take full ownership of the platform. You'll be responsible for: • Designing and maintaining ETL/ELT pipelines • Deploying ...

Senior Databricks Technical Engineer Up to £111,500 + 10% Bonus | UK Wide Hybrid | SC Clearance Ready to build data platforms that actually make an impact? We're looking for an experienced Senior Databricks Technical Engineer to join a growing enterprise data team where … play a key role in designing, building and evolving modern data platforms. This isn't a role for someone looking to learn Databricks-you'll be the technical expert from day one, helping shape the platform while mentoring other engineers. What you'll be doing Building enterprise-scale ...

domain specialised architecture position focused on Programme Management, PMO systems, Project Controls processes and EPPM technology ecosystems. Working within the Digital and Data Solutions Group under the Programme Controls Technology capability, you will collaborate closely with project management, project controls teams, programme delivery leadership, enterprise architects and technology … partners to define scalable, secure and integrated solutions for: Cost Management, Schedule Management, Risk, Change and Contingency, Performance Reporting, Data Governance and Integrated Analytics. You will act as a technical authority for project controls platforms, ensuring solutions align with enterprise architecture principles, programme delivery strategy and best practice ...

applications without this cannot be considered) Core Engineering: Strong backend development skills (Python, Go, or C++) with solid API design and data pipeline experience AI Tooling: Hands‐on experience integrating LLMs, handling vector databases, and building robust chunking/embedding strategies Secure Architecture: Deploying LLM pipelines ...

architectural leadership by leading through example. Your responsibilities will include: Lead the development and implementation of a modern cloud foundation and data platform that is robust, scalable, fully automated, secure, and can support the growth of the business. Build Scalable Architectures: Leverage modern technologies to design and implement … build secure RESTful and GraphQL APIs, ensuring seamless integration with core banking systems (e.g., Mambu) and external services like Open Banking platforms. Data Engineering and Analytics: Work closely with data teams to define robust data pipelines and scalable cloud-based data platforms ...

Data Scientist (AI & Machine Learning) – Renewable Energy & Emerging Technologies London/Hybrid About the Role We're looking for an AI-focused Data Scientist to join our Engineering & Development team, helping transform how renewable energy projects are identified, evaluated and developed. You'll work … technologies can improve the way renewable energy projects are developed. You'll also have the opportunity to contribute to new technology areas including data centres, onshore wind and future energy infrastructure. The organisation sees AI and advanced analytics as a key enabler of future growth and is looking ...

looking for a hands-on Data Product Manager to own and personally lead the development of our predictive market modelling solutions for the Animal Health division. This is a player-coach role: you will set the product direction for our sales and market models and roll up your … yourself. You’ll be the technical and product owner for forecasting solutions that directly shape commercial decision-making – turning messy internal and external data into models the business trusts and acts on. The role suits someone equally comfortable in a stakeholder workshop and in a notebook, who wants ...

Integration Engineer with active UK SC clearance for a contract position in London. The role focuses on backend development, API design, and data pipelines, with hands-on work integrating LLMs and vector databases in secure, classified environments. You will work on self-hosted, containerised models (Llama, Mistral ...

deep technical engine behind their internal AI deployment team. Where the AI Deployment Strategist scopes and builds agents, you own the infrastructure underneath data pipelines, system integrations, and everything that ensures deployed AI solutions work reliably at scale. You will take prototypes to production, ensure all business systems … storage supporting current and future AI deployments Troubleshoot integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft ...

need proven experience in the following: Hands-on experience with Splunk in a SOC environment Cribl experience for log management and data pipeline optimisation Security monitoring, threat detection and incident response Working within a security operations centre at a mid-senior level SC clearance held or ability ...

patterns. Contribute to technical strategy and architecture decisions, working closely with the Head of Engineering. Maintain secure, performant, and maintainable systems, ensuring reliable data pipelines as we scale. Engage closely with product workflows to ensure systems built solve the right problems. Job Requirements: Experience in software engineering, ideally ...

internal tooling. Provide sound technical recommendations to the Senior Manager, clients, and internal stakeholders, evaluating options and trade-offs. Own technical design decisions, data models, and integration patterns; document and communicate them so others can build against them. Help define technical standards and pragmatic build/buy/… adopt AI-driven solutions effectively and responsibly. Hands-on development (full stack) Write production-quality code across front end, back end, and data layers. This is and remains a genuinely hands-on role. Build and integrate APIs, data pipelines, and tooling that turn actuarial and financial ...

Astronomer empowers data teams to bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data … deployed. Your work will directly influence how we build Astro, Observe and our IDE product, as well as how global organizations orchestrate data pipelines at scale—making them faster, more reliable, and easier to manage. If you’re driven by impact, excited by scale, and ready to work ...

e.g., Python, Python API, Java,SQL ) and engineering frameworks Work with distributed systems and cloud platforms to support scalable, reliable deployments Assist with data preparation, prompt configuration, and model integration for AI-driven solutions Troubleshoot and resolve technical issues during development and deployment Client Delivery Support Work alongside … production, ensuringsolutions are scalable, supportable, and can be transitioned effectively to production engineering teams Skill Requirements Bachelor's degree in Computer Science, Engineering, Data Science, or related field Total 15+ year of Experience in Client Facing - AI Implementation Cloud platforms Application interfacesas Solution Architect. 5+ years of proficient ...

company that’s tackling one of the biggest challenges in energy today. They’re building a new approach to natural exploration, using advanced data, geoscience, and cloud tech to drive smarter decisions. They’re now looking for a Senior Data Platform Engineer to take ownership … their growing platform and help scale it as the business expands globally. What you’ll be doing This Senior Data Platform Engineer role sits at the heart of a multidisciplinary team of scientists, analysts, and engineers. You’ll work closely with them to turn complex workflows into robust ...

innovative geospatial solutions within a risk and analytics environment. This is a hands-on role focused on building production-grade geospatial services, spatial data pipelines and interactive mapping solutions. You'll work closely with architects, product leaders and engineering teams to develop scalable GIS platforms while making … delivery. What You'll Be Doing Designing and developing geospatial services and APIs that underpin business-critical analytics products. Building and maintaining spatial data pipelines for ingesting, transforming and validating large geospatial datasets. Applying advanced GIS expertise across coordinate systems, spatial analysis, topology, cartography and data ...

Oriented Programming (OOP) to build intelligent, scalable automation solutions, while actively driving the adoption of AI-powered testing tools, autonomous test agents, and data-driven quality insights to modernise the testing lifecycle for our clients. Key Responsibilities Embeds Shift Left & AI-Assisted Quality Practice by engaging in quality … Python. Leverages Cloud & AI-Integrated Test Infrastructure. Operates within AWS/GCP environments to execute cloud-native and distributed test execution, manage test data pipelines and monitor logs and quality signals within CI/CD pipelines. Drives Agentic AI in Testing. Designs, implements and experiments with Agentic ...

maintenance. Produce technical designs, testing strategies and implementation plans. Partner with Finance stakeholders to understand business needs and translate them into process workflows, data requirements and specifications. Build reusable solutions and identify opportunities for cross-divisional collaboration. Contribute to data modelling and curation. Basic Qualifications Bachelor … Computer Science, Mathematics, Electrical Engineering or related discipline. 6+ years of software development experience. Strong analysis and design skills, plus knowledge of data structures, algorithms and performance-oriented design. Strong communication skills with both technical and business audiences, and comfortable working with multiple global stakeholders. Expert Knowledge ...

Requirements (Must Have Skills) 6 to 8 years of experience in data software engineering with a strong focus on Azure technologies Expertise in Azure Databricks development , including efficient handling of large datasets Strong working knowledge of Azure Data Factory for building complex ETL workflows Experience with … performance optimization of data pipelines and distributed processing setups Proficiency in Scala development (hands-on experience highly desirable) Solid understanding of Azure services relevant to data engineering and cloud architecture Strong problem-solving ability with effective communication skills for team collaboration ...

Ports North is seeking a Data Software Engineer to design and build Azure Databricks-based data solutions in London, hybrid working mode. In this role, you will design and build data engineering solutions leveraging Azure Databricks and related Azure services, enabling efficient processing … transformation of large-scale datasets. You will collaborate with cross-functional teams to ensure the delivery of optimized, reliable, and scalable data pipelines aligned with business requirements. #J-18808-Ljbffr ...

Senior Forward Deployed Engineer (FDE) London Financial Data Platforms Join one of the fastest growing fintech scale‐ups in London who are operating at the heart of the investment lifecycle, working with some of the biggest names in finance across the globe. This role suits a senior software … business problems into platform structures; debug across the full stack and pinpoint root causes Work on cutting edge solutions Build and configure client data pipelines, integrations, queries, workflows, and config‐driven deployments Mentor engineers and graduates; set quality and engineering standards through code review and design guidance What ...

Advanced Resource Managers Ltd. is seeking an experienced Python Data Engineer for a 3-6 month initial contract in London with hybrid work (1-2 days on site). The role focuses on designing, developing and maintaining scalable data platforms and pipelines used by quantitative researchers ...