1 to 25 of 922 Permanent Data Pipeline Jobs in London

Senior Data Engineer - (Azure/Databricks) page is loaded Senior Data Engineer - (Azure/Databricks) Apply locations London - Scalpel time type Full time posted on Posted 15 Days Ago job requisition id REQ05851 This is your opportunity to join AXIS Capital - a trusted global provider of specialty lines insurance and reinsurance. We stand apart for our … civil union status, family or parental status, or any other characteristic protected by law. Accommodation is available upon request for candidates taking part in the selection process. Senior Data Engineer (Azure/Databricks Job Family Grouping: Chief Underwriting Officer Job Family: Data & Analytics Location: London How does this role contribute to our collective success? The Data … services. Implement end-to-end data pipelines, ensuring data quality, data integrity and data security. Troubleshoot and resolve data pipeline issues while ensuring data integrity and quality. Implement and enforce data security best practices, including role-based access control (RBAC), encryption, and compliance with industry More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … data engineering projects. You will work closely with cross-functional teams and contribute to the strategic direction of our data initiatives. RESPONSIBILITIES Data Pipeline Development: Lead the design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such More ❯

a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets. Key Responsibilities : Data Pipeline Development & Optimisation : Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data … technical solutions. Communicate technical concepts effectively to both technical and non-technical audiences. Participate in code reviews and knowledge sharing sessions. Automation & DevOps: Implement automation for data pipeline deployments and other data engineering tasks. Work with DevOps teams to implement and Build CI/CD pipelines, for environmental deployments. Promote and implement DevOps best practices. … Azure Blob Storage, and Azure SQL Database. Experience working with large datasets and complex data pipelines. Experience with data architecture design and data pipeline optimization. Proven expertise with Databricks, including hands-on implementation experience and certifications. Experience with SQL and NoSQL databases. Experience with data quality and data governance More ❯

a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets. Key Responsibilities : Data Pipeline Development & Optimisation : Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data … technical solutions. Communicate technical concepts effectively to both technical and non-technical audiences. Participate in code reviews and knowledge sharing sessions. Automation & DevOps: Implement automation for data pipeline deployments and other data engineering tasks. Work with DevOps teams to implement and Build CI/CD pipelines, for environmental deployments. Promote and implement DevOps best practices. … Azure Blob Storage, and Azure SQL Database. Experience working with large datasets and complex data pipelines. Experience with data architecture design and data pipeline optimization. Proven expertise with Databricks, including hands-on implementation experience and certifications. Experience with SQL and NoSQL databases. Experience with data quality and data governance More ❯

class opportunity to help shape our organisation for the next stage of its journey. To drive towards this ambition, we are seeking a motivated individual to join our Data Platform team and support Aztec's new technology strategy using Azure Databricks. You will lead our Data Engineering capability and collaborate with others passionate about solving business … problems. Key responsibilities: Data Platform Design and Architecture Design, develop, and maintain a high-performing, secure, and scalable data platform, leveraging Databricks Corporate Lakehouse and Medallion Architectures. Utilise our metadata-driven data platform framework combined with advanced cluster management techniques to create and optimise scalable, robust, and efficient data solutions. Implement comprehensive … Background in cloud platforms and data architectures, such as Corporate DataLake, Medallion Architecture, Metadata Driven Platform, Event-driven architecture. Proven experience of ETL/ELT, including Lakehouse, Pipeline Design, Batch/Stream processing. Strong working knowledge of programming languages, including Python, SQL, PowerShell, PySpark, Spark SQL. Good working knowledge of data warehouse and data More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … modelling, ETL (Extract, Transform, Load) processes, and big data technologies it becomes possible to develop robust and reliable data solutions. RESPONSIBILITIES Data Pipeline Development: Design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such as databricks More ❯

Job title: Sr. Data Engineer Will the role be 100% remote, hybrid or 100% office? Hybrid – 3 days/week If the role is hybrid/office based specify location: London, UK Duration of assignment: 3 months contract InsideIR35 Role description: As a Senior Data Engineer, you will play a crucial role in designing, developing, and … maintaining scalable data pipelines, ensuring high-quality data is available for decision-making across the business. You’ll collaborate closely with data architects, product teams, analysts, and data scientists to implement data solutions that power analytics and reporting. The ideal candidate will have strong experience in data engineering … You’ll be part of a dynamic and innovative team, working on cutting-edge technologies to deliver impactful data products for our retail chain Key responsibilities: ETL Pipeline Development: Develop, optimize, and maintain ETL pipelines to efficiently extract, transform, and load data from various sources, ensuring high data quality. Monitor and troubleshoot production More ❯

Job title: Sr. Data Engineer Will the role be 100% remote, hybrid or 100% office? Hybrid – 3 days/week If the role is hybrid/office based specify location: London, UK Duration of assignment: 3 months contract InsideIR35 Role description: As a Senior Data Engineer, you will play a crucial role in designing, developing, and … maintaining scalable data pipelines, ensuring high-quality data is available for decision-making across the business. You’ll collaborate closely with data architects, product teams, analysts, and data scientists to implement data solutions that power analytics and reporting. The ideal candidate will have strong experience in data engineering … You’ll be part of a dynamic and innovative team, working on cutting-edge technologies to deliver impactful data products for our retail chain Key responsibilities: ETL Pipeline Development: Develop, optimize, and maintain ETL pipelines to efficiently extract, transform, and load data from various sources, ensuring high data quality. Monitor and troubleshoot production More ❯

Job title: Sr. Data Engineer Will the role be 100% remote, hybrid or 100% office? Hybrid – 3 days/week If the role is hybrid/office based specify location: London, UK Duration of assignment: 3 months contract InsideIR35 Role description: As a Senior Data Engineer, you will play a crucial role in designing, developing, and … maintaining scalable data pipelines, ensuring high-quality data is available for decision-making across the business. You’ll collaborate closely with data architects, product teams, analysts, and data scientists to implement data solutions that power analytics and reporting. The ideal candidate will have strong experience in data engineering … You’ll be part of a dynamic and innovative team, working on cutting-edge technologies to deliver impactful data products for our retail chain Key responsibilities: ETL Pipeline Development: Develop, optimize, and maintain ETL pipelines to efficiently extract, transform, and load data from various sources, ensuring high data quality. Monitor and troubleshoot production More ❯

to technology, delivering bespoke software solutions and expert advice. Our clients are increasingly looking to us to help them make the best use of their data. In building data platforms and pipelines, our data engineers create the foundation for diverse data & analytics solutions, including data science and AI. They build data lakes and warehouses, create the processes to extract or access operational data, and transform siloed datasets into integrated data models that allow insight into business performance and problems or training of ML models. These are hands-on, client-facing roles, with openings at senior or lead level to suit your experience. You may be … or solving tough engineering challenges. You'd also be expected to spend some time on-site with clients in the London area on an ad-hoc basis. Our data engineers combine a strong software engineering approach with solid data fundamentals and experience with modern tools. We're technology agnostic, and we're open minded when it More ❯

A capabilities and maintain them. Partnerships with other teams such as Supply, Manufacturing & R&D are also key to ensure process alignment and shared value creation. As a Data Engineer in the Commercial team, your key responsibilities are as follows: 1. Technical Proficiency: Collaborate in hands-on development using Python, PySpark, and other relevant technologies to create and … maintain data assets and reports for business insights. Assist in engineering and managing data models and pipelines within a cloud environment, utilizing technologies like Databricks, Spark, Delta Lake, and SQL. Contribute to the maintenance and enhancement of our progressive tech stack, which includes Python, PySpark, Logic Apps, Azure Functions, ADLS, Django, and ReactJs. Support the implementation … and Databricks. Proficiency in working with the cloud environment and various platforms, including Azure, SQL Server. NoSQL databases is good to have. Hands-on experience with data pipeline development, ETL processes, and big data technologies (e.g., Hadoop, Spark, Kafka). Experience with DataOps practices and tools, including CI/CD for data pipelines. More ❯

Senior Data Engineer – London - Hybrid Mars is a family-owned business with more than $35 billion in global sales. We produce some of the world’s best-loved brands: M&M’s®, SNICKERS®, TWIX®, MILKY WAY®, DOVE®, PEDIGREE®, ROYAL CANIN®, WHISKAS®, EXTRA®, ORBIT®, 5TM, SKITTLES®, BEN’S ORIGINAL®, and COCOAVIA®. Alongside our consumer brands, we proudly take … A capabilities and maintain them. Partnerships with other teams such as Supply, Manufacturing & R&D are also key to ensure process alignment and shared value creation. As a Data Engineer in the Commercial team, your key responsibilities are as follows: 1. Technical Proficiency: Collaborate in hands-on development using Python, PySpark, and other relevant technologies to create and … and Databricks. Proficiency in working with the cloud environment and various platforms, including Azure, SQL Server. NoSQL databases is good to have. Hands-on experience with data pipeline development, ETL processes, and big data technologies (e.g., Hadoop, Spark, Kafka). Experience with DataOps practices and tools, including CI/CD for data pipelines. More ❯

Senior Data Engineer – London - Hybrid Mars is a family-owned business with more than $35 billion in global sales. We produce some of the world’s best-loved brands: M&M’s®, SNICKERS®, TWIX®, MILKY WAY®, DOVE®, PEDIGREE®, ROYAL CANIN®, WHISKAS®, EXTRA®, ORBIT®, 5TM, SKITTLES®, BEN’S ORIGINAL®, and COCOAVIA®. Alongside our consumer brands, we proudly take … A capabilities and maintain them. Partnerships with other teams such as Supply, Manufacturing & R&D are also key to ensure process alignment and shared value creation. As a Data Engineer in the Commercial team, your key responsibilities are as follows: 1. Technical Proficiency: Collaborate in hands-on development using Python, PySpark, and other relevant technologies to create and … and Databricks. Proficiency in working with the cloud environment and various platforms, including Azure, SQL Server. NoSQL databases is good to have. Hands-on experience with data pipeline development, ETL processes, and big data technologies (e.g., Hadoop, Spark, Kafka). Experience with DataOps practices and tools, including CI/CD for data pipelines. More ❯

in the West End of London (links Piccadilly in the south to Oxford Street in the north) **Work Arrangement:** In-office only, five days a week. **Job Title:** Data Architect/Data Engineer **Experience Required:** 7-8 Years **Role Overview:** We are seeking a Senior Data Architect/Data Engineer to design … and implement scalable data solutions, including real-time and ETL/ELT data pipelines. Responsibilities include optimizing our data warehouse, creating Power BI reports, and developing AI-driven solutions using OpenAI APIs. **Key Responsibilities:** - Design and implement robust real-time data pipelines from diverse sources. - Maintain a high-performance data warehouse for reporting and analytics. - Develop insightful Power BI dashboards and reports for stakeholders. - Use Azure Data Factory, Azure Functions, and Logic Apps for data integration. - Employ Python and SQL for data transformation and analysis. - Leverage OpenAI APIs for AI/ML solutions. - Collaborate with teams to align data strategy More ❯

in the West End of London (links Piccadilly in the south to Oxford Street in the north) **Work Arrangement:** In-office only, five days a week. **Job Title:** Data Architect/Data Engineer **Experience Required:** 7-8 Years **Role Overview:** We are seeking a Senior Data Architect/Data Engineer to design … and implement scalable data solutions, including real-time and ETL/ELT data pipelines. Responsibilities include optimizing our data warehouse, creating Power BI reports, and developing AI-driven solutions using OpenAI APIs. **Key Responsibilities:** - Design and implement robust real-time data pipelines from diverse sources. - Maintain a high-performance data warehouse for reporting and analytics. - Develop insightful Power BI dashboards and reports for stakeholders. - Use Azure Data Factory, Azure Functions, and Logic Apps for data integration. - Employ Python and SQL for data transformation and analysis. - Leverage OpenAI APIs for AI/ML solutions. - Collaborate with teams to align data strategy More ❯

AWS Data Engineer (Must hold current SC) The Company: At Amber Labs, we are a cutting-edge UK and European technology consultancy that prioritises empowering autonomy, promoting experimentation, and facilitating rapid learning to provide exceptional value to our clients. Our company culture is centred around collaboration, where all colleagues, regardless of their role, work together to minimise risk … and shorten delivery times. Our team consists of highly-skilled cross-functional consultants, analysts, and support staff. Overview: We are looking for a highly skilled AWS Data Engineer to join our team. The ideal candidate will have expertise in designing, building, and managing scalable data pipelines on AWS. You will work closely with data scientists, analysts, and software engineers to implement efficient data solutions that support business intelligence and analytics needs. Responsibilities: Design, develop, and maintain scalable ETL/ELT pipelines using AWS services such as AWS Glue, Lambda, Step Functions, and Kinesis. Work with structured and unstructured data from multiple sources, ensuring efficient data ingestion More ❯

to technology, delivering bespoke software solutions and expert advice. Our clients are increasingly looking to us to help them make the best use of their data. In building data platforms and pipelines, our data engineers create the foundation for diverse data & analytics solutions, including data science and AI. They build data lakes and warehouses, create the processes to extract or access operational data, and transform siloed datasets into integrated data models that allow insight into business performance and problems or training of ML models. These are hands-on, client-facing roles, with openings at senior or lead level to suit your experience. You may be … or solving tough engineering challenges. You'd also be expected to spend some time on-site with clients in the London area on an ad-hoc basis. Our data engineers combine a strong software engineering approach with solid data fundamentals and experience with modern tools. We're technology agnostic, and we're open minded when it More ❯

provides its customers with clear advantage, and its technology-led service provides access to all major exchanges, order-flow management via screen, voice and DMA, plus award-winning data, insights and analytics. The Technology Department delivers differentiation, scalability and security for the business. Reporting to the COO, Technology provides digital tools, software services and infrastructure globally to all … design. The IT Group runs our enterprise-wide services to end users and actively manages the firm's infrastructure and data. Within IT, Marex Technology has established a Data team that enables the firm to leverage data assets to increase productivity and improve business decisions, as well as maintain data compliance. The Data Team encompasses Database Administration, Data Engineering, Data Analysis, Data Architecture, Data Intelligence and AI expertise. In recent years, they have developed a Data Lakehouse architecture, that is relied upon by different departments across the firm. Marex now seeks to strengthen its capabilities further and elevate the role of More ❯

Senior Analytics Engineer Location: London, Hybrid (minimum 2 days per week in office) Type: Full-Time, Permanent Reports to: Engineering Manager/Lead Data Engineer Salary: £55,000-£70,000 per annum Due to the nature of our work, candidates must be eligible for Security Clearance. Join Our Mission At QuantSpark, we're transforming how data … street brands, and premium consumer goods companies optimise inventory, enhance price competitiveness, and deliver personalised customer experiences. Asset Management: We build AI-powered investment decision support systems, alternative data integration platforms, and automated risk management solutions for prestigious asset management firms. Public Sector: We're trusted by multiple UK government departments and public sector organisations to deliver predictive … monitoring, and documentation standards that ensure our solutions are production-ready and maintainable. Mentor Rising Talent: Guide junior analytics engineers and data scientists, sharing your expertise in pipeline design, SQL optimisation, and Python development. You'll help build the next generation of analytics engineering talent at QuantSpark. Shape Client Solutions: Work directly with clients to understand their More ❯

About Quantum: Quantum is building next-generation AI and trading technologies that harness cutting-edge research and data science. As part of our rapid growth, we are seeking a highly skilled Senior Data Engineer to support our Data & AI team by designing and maintaining robust, scalable, and production-grade data systems. This … is greenfield project, allowing for full product ownership and key decision making. Role Overview: As a Senior Data Engineer at Quantum, you will be instrumental in building the infrastructure that powers our Data Science, AI and Trading tools. You will work closely with the Data and Technology teams to ensure data accessibility … in a fast-paced, research-driven environment and wants to be part of building world-class trading system capabilities from the ground up. Key Responsibilities: Develop and Maintain Data Pipelines: Design, build, and optimise scalable data pipelines to support AI research and production systems, particularly for unstructured, text-heavy and time-series based datasets. Data More ❯

About Quantum: Quantum is building next-generation AI and trading technologies that harness cutting-edge research and data science. As part of our rapid growth, we are seeking a highly skilled Senior Data Engineer to support our Data & AI team by designing and maintaining robust, scalable, and production-grade data systems. This … is greenfield project, allowing for full product ownership and key decision making. Role Overview: As a Senior Data Engineer at Quantum, you will be instrumental in building the infrastructure that powers our Data Science, AI and Trading tools. You will work closely with the Data and Technology teams to ensure data accessibility … in a fast-paced, research-driven environment and wants to be part of building world-class trading system capabilities from the ground up. Key Responsibilities: Develop and Maintain Data Pipelines: Design, build, and optimise scalable data pipelines to support AI research and production systems, particularly for unstructured, text-heavy and time-series based datasets. Data More ❯

Data Engineer - Leading Energy Company - London (Tech Stack: Data Engineer, Databricks, Python, PySpark, Power BI, AWS QuickSight, AWS, TSQL, ETL, Agile Methodologies) Company Overview: Join a dynamic team, a leading player in the energy sector, committed to innovation and sustainable solutions. Our client are seeking a talented Data Engineer to help build and optimise … our data infrastructure, enabling them to harness the power of data-driven insights to drive our business forward. Responsibilities: Design and develop a cutting-edge data warehouse capable of efficiently ingesting and organising large volumes of data from multiple sources. Champion best practices in data architecture governance, ensuring compliance with … security and privacy regulations. Implement automated, scalable data migration processes across various project phases. Conduct rigorous data quality assessments, employing cleansing and validation techniques as needed. Construct robust data pipelines for cleaning, transforming, and aggregating diverse datasets. Collaborate closely with software development and product teams to align data strategies with business objectives. More ❯

Data Engineer - Leading Energy Company - London (Tech Stack: Data Engineer, Databricks, Python, PySpark, Power BI, AWS QuickSight, AWS, TSQL, ETL, Agile Methodologies) Company Overview: Join a dynamic team, a leading player in the energy sector, committed to innovation and sustainable solutions. Our client are seeking a talented Data Engineer to help build and optimise … our data infrastructure, enabling them to harness the power of data-driven insights to drive our business forward. Responsibilities: Design and develop a cutting-edge data warehouse capable of efficiently ingesting and organising large volumes of data from multiple sources. Champion best practices in data architecture governance, ensuring compliance with … security and privacy regulations. Implement automated, scalable data migration processes across various project phases. Conduct rigorous data quality assessments, employing cleansing and validation techniques as needed. Construct robust data pipelines for cleaning, transforming, and aggregating diverse datasets. Collaborate closely with software development and product teams to align data strategies with business objectives. More ❯

Analytics Engineer Location: London, Hybrid (minimum 2 days per week in office) Type: Full-Time, Permanent Reports to: Engineering Manager/Lead Data Engineer Salary: £55,000-£70,000 per annum Due to the nature of our work, candidates must be eligible for Security Clearance. Join Our Mission At QuantSpark, we're transforming how data and … street brands, and premium consumer goods companies optimise inventory, enhance price competitiveness, and deliver personalised customer experiences. Asset Management: We build AI-powered investment decision support systems, alternative data integration platforms, and automated risk management solutions for prestigious asset management firms. Public Sector: We're trusted by multiple UK government departments and public sector organisations to deliver predictive … monitoring, and documentation standards that ensure our solutions are production-ready and maintainable. Mentor Rising Talent: Guide junior analytics engineers and data scientists, sharing your expertise in pipeline design, SQL optimisation, and Python development. You'll help build the next generation of analytics engineering talent at QuantSpark. Shape Client Solutions: Work directly with clients to understand their More ❯

Who We Are BCG Expand is a specialist firm providing unique data-driven business intelligence as a trusted advisor to senior executives across the world's leading financial services firms to help them to operate more effectively. Expand's activities are focused on data benchmarking in technology, operations, FinTech and market data, with expertise … our professional, supportive and entrepreneurial working culture that we all enjoy. What You'll Do We're looking for a highly skilled Principal Engineer with a passion for data, cloud technologies, and building robust, scalable solutions - who also brings strategic product thinking and cross-functional leadership . In this role, you'll not only lead the design and … implementation of the data infrastructure that powers advanced analytics and data science across our platforms, but also contribute to the strategic direction, roadmap, and execution of data products in collaboration with senior stakeholders. Architect & Build Scalable Data Solutions Collaborate closely with senior product stakeholders to understand data needs and architect More ❯