1 to 25 of 29 Data Pipeline Jobs in Central London

Data Engineer - £350PD - Remote Required Technical Skills Data Pipeline & ETL Design, build, and maintain robust ETL/ELT pipelines for structured and unstructured data Hands-on experience with AWS Glue and AWS Step Functions Implementation of data validation, data quality frameworks, and reconciliation checks Strong error handling, monitoring, and retry strategies in production pipelines Experience with incremental data processing patterns (CDC, watermarking, upserts) AWS Data Services Amazon S3: data lake architectures, partitioning strategies, lifecycle policies DynamoDB: data modeling, secondary ...

experienced Data Engineer to design, build, and optimise data solutions within the Microsoft Azure ecosystem. The role focuses on pipeline development, data modelling, governance, and supporting analytics teams with high-quality, reliable data. Key Responsibilities: Develop and maintain scalable data pipelines … data security and governance, including Row-Level Security (RLS) and compliance standards. Maintain documentation and metadata for data assets and pipeline architecture. Collaborate with analysts, data scientists, and stakeholders to deliver trusted data products. Provide technical support and troubleshoot production issues. ...

Factory , Databricks , and related Azure services. Build ETL/ELT processes to transform raw data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data … Azure SQL Database, Synapse Analytics). Strong hands-on expertise in Databricks , Delta Lake , and cluster management . Proficiency in SQL and Python for pipeline development. Familiarity with Git/GitHub and CI/CD practices. Understanding of data modelling , data governance , and security principles. ...

Data Engineering Technical Lead Global Investment Bank London - Hybrid Permanent - Excellent Package + Benefits We are working with one of the world's leading banking groups, who we have partnered with for 15 years. We are seeking an experienced Data Architect/EDM Developer/Data Engineering Lead to join their International Technology team in London. You will be a key part of the Architecture, Middleware, Data & Enterprise Services (AMD) division, driving data engineering, integration and automation initiatives across our clients EMEA banking and securities entities. This is a hands ...

Senior Data Engineer The Senior Data Engineer will play a crucial role in designing, implementing, and maintaining scalable data pipelines and infrastructure. This position is ideal for those with strong technical expertise and a passion for working in the Insurance/Financial services industry. … Client Details Senior Data Engineer The employer is a medium-sized organisation operating in the F sector. They focus on delivering innovative solutions and maintaining a strong reputation for excellence in analytics and data-driven decision-making. Description Senior Data Engineer Develop and maintain ...

Role title: Snr Data Engineer Security Clearance required: SC Clearance IR35 Status: Inside IR35 Pay Rates: £500/d - £525/d Contract length: 6 months Base Location: London - 3/d a week Skills required: - Azure/Azure Data bricks - ADF/ETL - Pyspark/… large data sets, and complex data pipelines. Understanding of Data Architecture and Design, and Data pipeline optimisation. Proven expertise with Data bricks, including hands on implementation experience and certifications. Reasonable Adjustments: Respect and equality are core values to us. ...

Data Engineer | Hybrid | London | Databricks | Azure | 85k I'm working with a global powerhouse that's been setting the standard for excellence for over 60 years. With more than 1,000 projects delivered worldwide and a combined value exceeding $150 billion, they've earned a reputation … organisation where world-class projects meet an environment that prioritises your well-being and career development. I'm looking for a Data Engineer who thrives on innovation and loves tackling complex data challenges. If building scalable, cloud-based solutions excites you, this is your chance ...

Data Engineer - 14-Week Contract (Outside IR35) Likely to Extend Start Date: 12th January Rate: £350 per day Location: Remote (UK-based) Interview: Immediate - Offer before Christmas We are seeking an experienced Data Engineer to join a 14-week project focused on building robust data … REST APIs , PIM systems, and potentially SAP. Ensure data quality, governance, and lineage tracking throughout the project. Required Skills ETL/ELT pipeline design and data validation frameworks. Advanced Python (pandas, numpy, boto3) and SQL (complex queries, optimisation). Experience with AWS Glue , Step Functions ...

Python programming including object-oriented design, asynchronous programming, error handling, and writing clean, maintainable code. Experience with key libraries including Pandas, NumPy for data manipulation, requests and APIs for integrations, asyncio for concurrent processing, and building robust automation scripts with proper logging, testing (pytest), and documentation. AI & Machine … when RPA vs. API integration vs. AI solutions are most appropriate, and experience building hybrid automation solutions combining multiple technologies. Data Engineering & Pipeline Development - Strong skills in building data pipelines for AI/automation solutions including data extraction, transformation, and loading (ETL). ...

exciting digital transformation where tech solutions will help enable the business to scale. This will be working across both Software and Data Engineering teams. They build .Net Solutions and use SAS, MicroStrategy and Power BI for Data Analytics/Insights. As they move towards an Agile … engineering and software engineering projects Experience of workflows, orchestrations and data/system integration DevOps and end to end data pipeline Prior experience of Python scripting Previous .Net Development experience nice to have Previously designed solutions to move solutions from on-premise to Cloud ...

130k + Equity Paddington based (4-5 days per week in-office) Tech environment: Python | PyTorch | TensorFlow | scikit-learn | NLP | LLMs | Data Engineering If you wish to keep your CV/data private, feel free to WhatsApp your details/… etc.). Experience deploying models in production environments (Docker, cloud services). Exposure to data engineering or working alongside data pipeline teams. A genuine passion for AI innovation and real-world applications in health, petcare, or automation. TO BE CONSIDERED Please either apply online ...

Description: Data Engineer. 1 year Contract. Remote £490 per day. Inside IR35 Our Central Government client is looking for an experienced Data Engineer to work on a brand new AI-focused Digital project in a GDS environment. Technical Responsibilities Build and maintain data pipelines … Azure (ADF, Databricks, Key Vault). Develop ingestion workflows for APIs, structured data, and CSV sources. Implement Infrastructure as Code (Bicep) for Dev/Test environments. Deliver Bronze and Silver layer transformations using Python and SQL. Create data validation and data quality reporting processes. ...

Founding Solutions Data Architect London/WFH to £115k Are you a data technologist with start-up experience looking for your next opportunity? You could be progressing your career, in a founding position at a tech start-up that is producing an AI native data pipelining platform. As the Founding Solutions Data Architect you'll play a vital role in helping client data teams to adopt and scale the company's platform. You'll be responsible for a broad range of activities including Solution Design and Architecture, Technical Onboarding ...

develop, and maintain core components of the SaaS platform, including: Backend: Python, PostgreSQL Frontend: TypeScript, NextJS/NodeJS Integrate AI tools, APIs, and data pipelines to enhance platform intelligence. Uphold engineering best practices such as testing, CI/CD, and secure coding standards. Collaborate with the Head ...

About the RoleWe are seeking a Databricks Data Engineer with strong expertise in designing and optimising large-scale data engineering solutions within the Databricks Data Intelligence Platform. This role is ideal for someone passionate about building high-performance data pipelines and ensuring … robust data governance across modern cloud environments. Key Responsibilities Design, build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data. Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies. Conduct performance ...

Senior Manager We are looking for a Senior Manager with deep Palantir Foundry expertise to lead the design and delivery of production-grade data and AI solutions. You will shape end-to-end architectures, lead multidisciplinary teams and work directly with senior client stakeholders to turn complex data … platform. Skills and experience Significant hands-on experience delivering Palantir Foundry solutions in complex client environments. Deep Foundry technical expertise across the full stack: Pipeline Builder, Ontology, Workshop, OSDK, Code Repositories, Actions and AIP or agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency ...

leading Financial Services client in London is undergoing a major data transformation and is hiring a Data Engineering Manager on a permanent basis. The role offers a base of £85,000, strong benefits, and flexible hybrid/remote working. Were looking for a hands-on leader … from a forward-thinking, modern data environment with a strong foundation in Data Engineering and experience managing a squad of 47 Data Engineers. Youll bring solid SQL skills, cloud expertise (preferably Azure), and familiarity with tools such as Snowflake, Databricks, ADF, Python, and GenAI. ...

engineering along with apis, optimising the application for performance, arbitrage bots and more. Requirements : 3+ years experience with backend Python : Including async systems, data pipelines, or protocol-level integrations. 1+ year of recent Rust experience : With the ability to read, modify, and contribute to Rust-based components. DevOps ...

Databricks Data Engineer London | Senior Manager | Up to £100K + Bonus Ready to take your data engineering career to the next level? Join a global consultancy on a major transformation project within the insurance domain. This is your chance to work with cutting-edge technologies, influence … make a real impact in a collaborative, forward-thinking environment. Why This Role? Be part of a high-profile project driving innovation in data and analytics. Work with a global leader in digital transformation. Enjoy senior-level responsibilities, clear progression, and exposure to decision-makers. Competitive package ...

Data Scientist Location: Canary Wharf Employment Type: Contract, 2 months, strong chance of extension About the Role Morgan Hunt are working with a leading government organisation to recruit a GIS Data Scientist who can blend spatial analysis, advanced analytics, and problem-solving to turn geospatial data into actionable insights. You'll work with large, complex datasets, build predictive models, and support data-driven decisions across the organisation. If you love maps, patterns, and answering real-world questions with data, this role has your name all over it. Key Responsibilities Acquire ...

Senior Data Engineer (GenAI) - Leading Tech Company | London (Hybrid) Hybrid: 1 day in the office/4 days WFH Permanent role only Represented by Robert Half Limited Please note: No visa sponsorship on offer, No B2B,B2C, Fully remote workers or contractors on offer. The Opportunity … looking for a Senior Data Engineer with hands-on GenAI experience to join a forward-thinking technology business that's building data-driven solutions with real-world impact. This is a highly visible, senior-level role where you'll take ownership of data pipelines ...

role: Own and manage the cloud infrastructure, ensuring scalability, resilience, and security. Implement and maintain high-availability architectures across critical services and data layers. Design and optimise … infrastructure to support AI-powered workloads and data pipelines. Lead cost optimisation initiatives across the platform. Manage the end-to-end deployment pipeline to ensure fast, reliable, and low-friction releases. Maintain, optimise, and automate CI/CD processes. Ensure zero-downtime deployment strategies and effective rollback ...

role: Own and manage the cloud infrastructure, ensuring scalability, resilience, and security. Implement and maintain high-availability architectures across critical services and data layers. Design and optimise … infrastructure to support AI-powered workloads and data pipelines. Lead cost optimisation initiatives across the platform. Manage the end-to-end deployment pipeline to ensure fast, reliable, and low-friction releases. Maintain, optimise, and automate CI/CD processes. Ensure zero-downtime deployment strategies and effective rollback ...

delivery and oversight of technology supporting AML and Financial Crime functions, including: Owning the system architecture of AML platforms (rule engines, alerting systems, data pipelines). Designing scalable, secure and high-performance systems for transaction monitoring and analytics. Translating regulatory and audit requirements (KYC, CDD, CRR) into actionable … technical logic. Managing integrations with third-party data sources (PEP lists, sanctions vendors, screenings). Acting as a technical intermediary between compliance stakeholders and platform engineering teams. Conducting detailed data analysis in line with business and regulatory needs. Performing feasibility assessments, estimating resources, timelines and system ...

Role title: Snr Data Engineer Security Clearance required: SC Clearance IR35 Status: Inside IR35 Pay Rates: £500/d - £525/d Contract length: 6 months Base Location: London - 3/d a week Skills required: - Azure/Azure Data bricks - ADF/ETL - Pyspark/… Scala Experience: Experience working on large data sets, and complex data pipelines click apply for full job details ...