51 to 75 of 176 Data Pipeline Jobs in London

Data Engineer Paddington, London (Hybrid) Overview Data and Analytics is foundational to our Petcare OGSM and will drive our transformation to a business that is powered by data. To deliver on this ambition set by this OGSM we require the very highest level of technical/… engineering expertise within Global Petcare Data & Analytics. There is a need to expand this high performing team of creative, skilled individuals – to help build out new capabilities and bring fresh ideas to the table. Key Responsibilities Engineer and orchestrate data flows & pipelines in a cloud environment … in a cloud environment (Azure, AWS, GCP) Experience in at least one of: Python (or similar), SQL, PySpark Experience in building data pipeline/ETL/ELT solutions Ability and strong desire to research and learn new technologies and languages Interest in the contemporary data more »

Leading eCommerce client are now searching for a Lead Data Engineer to play a pivotal role in delivering their innovative data strategy. This role will shape our client’s data function, delivering end to end data requirements and designing data pipelines/warehouses to ensure data is streamlined and enhances overall business success. The role: Design, develop and test an array of data migration processes to streamline the conversation from existing system to a new Snowflake infrastructure. Manage the end-to-end data lifecycle, enabling the use of data to influence overall business decisions. Streamline our client’s data processes to ensure that customer journeys are personalised. Ensure the accuracy and reliability of data. Collaborate on business projects, advising on data and data more »

Job Title: Data Engineer Job Type: Full Time, Permanent Working location: London, Hybrid Role Purpose At Travelex we are developing modern data services, which will be at the heart of our relationship with our customers. Our data architecture is becoming more service-oriented and … for you to solve. Our work is fast-paced although our team remains collaborative and flexible even when delivering fast. Your role as data engineer has a wide remit: working with product owners, subject matter experts and managers across the company, to design and deliver new components of … to the traditional data engineering type! Significant hands-on experience with AWS, especially AWS services focused on data flow, pipelines, data transformation, storage and streaming. Excellent data engineering skills, for example with SQL, Python, DBT and Airflow. Good understanding of service-oriented more »

and Oaktree Capital. Find out more about us by visiting our website at: http://www.hartreepartners.com/JOB OVERVIEW: As a Senior Data Scientist at Hartree, you will play a pivotal role in shaping our data strategy and driving the development of advanced machine learning … time series forecasting solutions, specifically focused on the European Power Markets. You will collaborate with cross-functional teams to design, implement, and optimize data pipelines and predictive models that inform trading decisions and enhance operational efficiency. JOB RESPONSIBILITIES: Market Data Analysis: Conduct comprehensive analysis of data … Python and its data science libraries (e.g., Pandas, NumPy, Scikit-Learn, TensorFlow, PyTorch). Solid understanding of ML and data pipeline architectures and best practices. Experience with big data technologies and distributed computing (e.g., Spark, Hadoop) is a plus. Proficient in SQL and more »

Data Engineer Location: London – Hybrid Employment: Full Time All applicants must hold a DV (Developed Vetting) clearance to apply. The Opportunity: For the Ministry of Defence (MOD), maintaining National Security is a 24-hour-a-day, 365-day-a-year mission. For this reason, it partners with this … Big 4 Firm to support its multidisciplinary AI & Cloud Data Engineering Capabilities. The Role: As an AI & Cloud Data Engineering practitioner, you’ll build large-scale data pipelines and infrastructure, utilising data integration tools to optimise complex data solutions. Collaborating … with high-performing Data Science and ML Engineering teams to drive innovation and insight through AI and Data across the Defence & Security sector. Required Skills/Experience: British Passport Holder with active DV (Developed Vetting Clearance) End-to-end development with data pipelines, ETL more »

consultancy client. The successful candidate will be responsible for designing, developing, and maintaining KDB databases, as well as creating and optimizing real-time data analytics solutions. You will work closely with the technology team to enhance our data management systems, ensure data accuracy and … integrity, and help us make data-driven decisions. Key Responsibilities: Design and develop KDB databases and data processing systems to support real-time data analysis and reporting. Optimize database performance, troubleshoot issues, and implement improvements as needed. Collaborate with data analysts and … other teams to understand data requirements and develop efficient data retrieval processes. Write and maintain KDB Q scripts, ensuring code is well-documented and follows best practices. Monitor data quality and accuracy, implementing data validation and cleansing processes. Develop and maintain data more »

Vertex Search is representing a fast-growing financial services firm that is now looking to make a Senior Data Engineering hire, the first role of its kind into the business with huge scope for impact and influence. The nascent data engineering team will collaborate with internal … teams around the business with a focus on data engineering, pipelines, tooling and services. Key Responsibilities: Data Engineering: Design and develop data services across various business functions Data Pipelines: Design and develop data pipelines for extracting, transforming, and loading data into appropriate storage systems. Data Tools and Services: Develop data tools and services to discover, understand, and utilize data effectively. Best Practice: Advise on and contribute to the development of data governance and management practices. Collaborate across teams to understand more »

on delivering quality services and solutions. Able to distill complex technical challenges to actionable and explainable decisions. Having in-depth knowledge on Azure data components with this feature to provide solutions and necessary recommendations on the architecture. Inspire teams by building consensus and mediating compromises when necessary. Demonstrate … technologies and developing a vision of their suitability and role as a new product within a platform & product landscape. Having knowledge on the data components such as Azure datafactory, data lake, Azure SQL DB, Synapse and visualization with expertise skills in terms of performance tuning and … advanced troubleshooting. 8+ years of experience in data architecture, analytics, design and development Good working knowledge in trouble shooting data warehouse/analytical systems using Azure Synapse. Strong, in-depth experience in data modelling and experience with business intelligence Design and develop batch and more »

Senior Azure Data Engineer - 6 Month Contract - Inside IR35 This is an Azure Data Engineer 6 month contract position for a financial services end client helping them on delivering and implementing new data pipelines. You will need prior financial services experience as well as … having strong Azure Data Factory and Databricks experience. Work Involved: Implementing and delivering data pipelines Providing mentorship to junior members of the team Working with the Platform team defining capabilities for the lake … framework Developing business conceptual data models Producing new frameworks using design patterns and data ingestion Requirements: Strong Data Pipeline experience Databricks Azure Data Factory Spark Azure cloud Financial Services If you're interested get in touch ASAP more »

Job Title: Data Operations Analyst Company: Global Asset Manager Location: City of London, 3 days per week in office Salary: £85,000 per annum + Benefits About the Company: Join a prestigious global asset manager with over $200 billion in assets under management. Our firm is renowned for … excellence. Based in the heart of London's financial district, you'll be part of a dynamic team shaping the future of financial data operations. Role Overview: As a Data Operations Analyst, you will play a pivotal role in implementing a cutting-edge global platform aimed … at centralizing data management processes. Leveraging your expertise in R, Python, SQL, and Enterprise Data Management (EDM), you'll contribute to streamlining data workflows and optimizing data quality across the organization. Key Responsibilities: Collaborate with cross-functional teams to understand data more »

Data Engineer £80,000-£95,000 London, Hybrid Permanent I am seeking an experienced Data Engineer who is passionate about designing, optimising, and maintaining AI-native data processing pipelines. Your role will involve tackling complex challenges with robust data pipelines across diverse … sources, enhancing the quality of data for our systems, and readiness to meet the challenges of scale. Responsibilities: Ensure the secure handling of data through end-to-end encryption, data isolation, and other best practices. Design robust data processing and indexing pipelines … handling large amounts of data across several modalities, ensuring robustness and timeliness. Curate and manage large-scale data ingestion and indexing pipelines, ensuring data quality and error handling. Manage cloud orchestration and data infrastructure, applying your expertise with various data more »

About the Role: I am looking for a talented Data Engineer to join our clients dynamic team on a 6-month contract basis. This is an exciting opportunity for a mid-level professional with 3-5 years of experience to make a significant impact on our data …/hybrid working options and a competitive day rate of £250-£400, falling inside IR35 regulations. Key Responsibilities: Design, develop, and maintain scalable data pipelines and ETL processes using AWS, Databricks, Python, Spark, and SQL. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions. Optimize and troubleshoot data workflows to ensure efficiency and reliability. Implement data quality and validation checks to ensure data integrity. Contribute to the development and maintenance of our data lake and more »

Data Engineer - London - £75,000 I am working with a hugely successful, global organisation who are looking to add a Data Engineer to their growing London based data team. This organisation have recently implemented a brand new data platform encompassing the full … Microsoft Azure Stack including leveraging Databricks in order to deliver on its goal of becoming truly data driven. In this role you will be responsible for the BI and data engineering lifecycle across the business supporting the global data needs of various business functions. … This role is a split between data engineering and data analytics and you will take a prominent role on a number of exciting projects. You will be joining a business that values its people at the centre of everything they do. They are looking to expand more »

Data Engineer - London/Hybrid - £70,000 FTC I am currently working with a well-established organisation within the education sector who are looking for a Data Engineer to join their established IT team. Their current project is based around the creation and implementation of a … brand new Azure Data Platform. You will take responsibility for the design, development and maintenance of the data platform utilising your experience with the Azure tech stack. This is an opportunity to join a team who are serious about the way that they use data and work within an organisation who are providing significant investment into all of their data teams. The teams itself are growing in size but the company are also investing in the technology that they use with an emphasis on Microsoft tech such as Azure Synapse, Databricks more »

consultancy client. The successful candidate will be responsible for designing, developing, and maintaining KDB databases, as well as creating and optimizing Real Time data analytics solutions. You will work closely with the technology team to enhance our data management systems, ensure data accuracy and … integrity, and help us make data-driven decisions. Key Responsibilities: Design and develop KDB databases and data processing systems to support Real Time data analysis and reporting. Optimize database performance, troubleshoot issues, and implement improvements as needed. Collaborate with data analysts and … other teams to understand data requirements and develop efficient data retrieval processes. Write and maintain KDB Q scripts, ensuring code is well-documented and follows best practices. Monitor data quality and accuracy, implementing data validation and cleansing processes. Develop and maintain data more »

Senior Data Engineer (Inside IR35) - £700 to 825 per day Location: Remote/Hybrid (London) Rate: £825 per day Contract: Inside IR35 Are you an experienced Data Engineer with a strong background in banking? Our client are seeking Senior Data Engineers to contribute to … innovative data projects within a leading financial institution. Key Responsibilities: Data Engineering: Design and implement data models, ETL processes, and data warehousing solutions. Programming: Utilize Python, Java, Scala, or GoLang to build and optimize data pipelines. Distributed Processing: Work with … Hadoop, Spark, and other platforms for large-scale data processing. Real-Time Data Streaming: Develop and manage pipelines using CDC, Kafka, and Apache Spark. Database Management: Handle SQL databases like Oracle, MySQL, or PostgreSQL. Data Governance: Ensure data quality, security, and compliance more »

Senior Data Engineer (Databricks) – Remote - 6 Months Contract (Initial) – Outside IR35 VIQU are seeking 4 Senior Data engineers (Databricks) on a contract basis to assist our client as they migrate over from legacy to AWS. The main focus will be to centralise the organisations data warehouses and consolidating/centralising a number of BI tools. Senior Data Engineers should have extensive experience working with Databricks on recent projects and be exposed to the latest features and toolsets of Databricks Role Responsibilities: Design and develop scalable data pipelines using Databricks. … Implement data processing workflows in Python and SQL. Manage and optimise data storage solutions on AWS. Ensure data quality and integrity across all stages of data processing. Develop and maintain documentation for data engineering processes. Utilise Databricks features, including Delta more »

Senior Data Engineer Python, SQL, Big Data Frameworks London, Geneva, Dubai - 4x a week (no flex) £110-120,000 + bonus Seeking a skilled Data Engineer to join a company specialising in quantitative trading and investment management. This role is pivotal in building and … maintaining the infrastructure that supports their cutting-edge trading strategies. Key Responsibilities: Design and manage scalable data pipelines. Integrate data from various sources and develop ETL processes. Optimize database systems for high performance. Work with data scientists and analysts to meet data needs. Automate data processing tasks to enhance efficiency. Implement data security measures and ensure compliance with industry standards. Requirements: Master's degree in Computer Science, Engineering, Mathematics, or related field from a top uni. Proven experience as a Data Engineer in the more »

GOAT Interactive is looking for a Junior Data Engineer to join our Data Engineering team, this role will be responsible for the development, testing, and maintenance of data pipelines and data storage systems on Google Cloud Platform (GCP). You will be … working with technologies such as Apache Airflow, BigQuery, Python, and SQL to transform and load large data sets, ensuring high data quality and accessibility for business intelligence and analytics purposes. Our business is growing quickly and with that so the scale of our data, we are currently working on several initiatives to improve our data solution and you would become an integral part of this. Key Responsibilities Design, construct, install, test, and maintain data pipelines. Ensure systems meet business requirements and industry practices for data integrity more »

About Tredence: - Tredence focuses on last mile delivery of insights into actions by uniting its strengths in business analytics, data science, and software engineering. The largest companies across industries are engaging with Tredence and deploying its prediction and optimization solutions at scale –empowering end users to improve decision … strategically and execute methodically. Develop and manage capacity and growth projection forecasts of the environment within budgets. Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional/non-functional business requirements. Drive optimization, testing and tooling to improve quality … years of relevant experience At least two years of experience building and leading highly complex, technical engineering teams. Strong hands-on experience in Data bricks Experience managing distributed teams preferred. Strong technical experience in large distributed systems, Data Warehousing, Data Lake at scale Project more »

Position: Software Engineer - Data & AWS We are seeking a Software Engineer to build and optimize data platforms leveraging AWS and key technologies like CDAP, Snowflake, and Databricks. You will design and implement robust and scalable data pipelines, ETL, and analytics systems in the cloud. … Responsibilities: Develop and enhance data pipelines, ETL processes using CDAP on AWS infrastructure. Build data integration flows to migrate large datasets into Snowflake data warehouse. Implement AWS infrastructure-as-code solutions for deployment automation. Instrument data pipelines and leverage monitoring for performance … tuning and reliability. Work with data scientists to optimize data workflows and models on Databricks. Follow security best practices for access control, encryption, and auditing across data platforms. Participate in architecture reviews and technology selections. Continuously monitor and improve data platforms for more »

Title: Head of Business Intelligence and Analytics Company Description: A leading company in the property sector with a strong focus on innovation and data driven decision making seeks a highly experienced professional to lead its Business Intelligence and Analytics team. Job Summary: We are searching for a talented … to spearhead our Business Intelligence and Analytics team. You will play a pivotal role in driving organizational growth and development by transforming raw data into actionable insights that inform strategic decision-making across various departments. Responsibilities: In this role, you'll find yourself equally split between hands-on … team with 50% devoted to hands-on tasks and 50% to management and leadership responsibilities' Oversee the construction and maintenance of a central data repository, ensuring alignment with organizational needs and data strategy. Collaborate with the data engineering team to gather and transform data more »

I'm looking for a Data Engineer with skills in Databricks, Azure Data Factory and SQL Server, for a Fixed Term Contract role until August 2025. It's with an international organisation headquartered in London, and this role will require you to be in their offices … near Victoria station 3 days per week. You will be joining a small cross-functional data team, where you will focus on the development of their newly-created Azure Data Platform - this will involve designing and developing ETL pipelines using Azure Data Factory, and … utilising Databricks to support the ongoing development of the cutting-edge Data Platform. You'll seek opportunities for improvement where possible, and will collaborate with both business and technology stakeholders to ensure that the solution is fit for purpose. You will also design and develop data more »

I'm looking for a Data Engineer with skills in Databricks, Azure Data Factory and SQL Server, for a Fixed Term Contract role until August 2025. It's with an international organisation headquartered in London, and this role will require you to be in their offices … near Victoria station 3 days per week. You will be joining a small cross-functional data team, where you will focus on the development of their newly-created Azure Data Platform - this will involve designing and developing ETL pipelines using Azure Data Factory, and … utilising Databricks to support the ongoing development of the cutting-edge Data Platform. You'll seek opportunities for improvement where possible, and will collaborate with both business and technology stakeholders to ensure that the solution is fit for purpose. You will also design and develop data more »

Senior Data Engineer Python, SQL, Big Data Frameworks London - 4x a week £80-90,000 + bonuses Seeking a skilled Data Engineer to join a company specialising in quantitative trading and investment management. This role is pivotal in building and maintaining the infrastructure that … supports their cutting-edge trading strategies. Key Responsibilities: Design and manage scalable data pipelines. Integrate data from various sources and develop ETL processes. Optimise database systems for high performance. Work with data scientists and analysts to meet data needs. Automate data processing tasks to enhance efficiency. Implement data security measures and ensure compliance with industry standards. Requirements: Master’s degree in Computer Science, Engineering, Mathematics, or related field from a top uni. Proven experience as a Data Engineer, preferably in the financial services or hedge more »