1 to 25 of 1,579 Data Pipeline Jobs in London

Senior Data Engineer - (Azure/Databricks) Axis Capital London, United Kingdom Apply now Posted 15 days ago Permanent Competitive Senior Data Engineer - (Azure/Databricks) Axis Capital London, United Kingdom Apply now This is your opportunity to join AXIS Capital - a trusted global provider of specialty lines insurance and reinsurance. We stand apart for our outstanding … civil union status, family or parental status, or any other characteristic protected by law. Accommodation is available upon request for candidates taking part in the selection process. Senior Data Engineer (Azure/Databricks Job Family Grouping: Chief Underwriting Officer Job Family: Data & Analytics Location: London How does this role contribute to our collective success? The Data … services. Implement end-to-end data pipelines, ensuring data quality, data integrity and data security. Troubleshoot and resolve data pipeline issues while ensuring data integrity and quality. Implement and enforce data security best practices, including role-based access control (RBAC), encryption, and compliance with industry More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … data engineering projects. You will work closely with cross-functional teams and contribute to the strategic direction of our data initiatives. RESPONSIBILITIES Data Pipeline Development: Lead the design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such More ❯

you share our passion for ideas and commitment to excellence, we want you to join us. To learn more visit alti-global.com. Job Description & Overview The Head of Data Engineering & Analytics will lead the development and execution of AlTi’s enterprise data engineering strategy, enabling the capture, transformation, storage and delivery of high-quality data across the firm’s global wealth, investment, corporate and asset management functions. This leader will architect and scale data engineering capabilities to support real-time and batch integration, reporting, and advanced analytics. This role reports to the CTO and will be a key member of the Global Technology Solutions leadership team. In this hands-on leadership … code, automated testing, release and version control and system observability for data pipelines. Establish metrics and KPIs and identify and deploy tools to measure data pipeline health, data quality, timeliness and accuracy, team performance, cost-effectiveness, and business impact. Actively mentor and grow talent within the team while fostering a collaborative and outcome More ❯

a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets. Key Responsibilities : Data Pipeline Development & Optimisation : Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data … technical solutions. Communicate technical concepts effectively to both technical and non-technical audiences. Participate in code reviews and knowledge sharing sessions. Automation & DevOps: Implement automation for data pipeline deployments and other data engineering tasks. Work with DevOps teams to implement and Build CI/CD pipelines, for environmental deployments. Promote and implement DevOps best practices. … Azure Blob Storage, and Azure SQL Database. Experience working with large datasets and complex data pipelines. Experience with data architecture design and data pipeline optimization. Proven expertise with Databricks, including hands-on implementation experience and certifications. Experience with SQL and NoSQL databases. Experience with data quality and data governance More ❯

a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets. Key Responsibilities : Data Pipeline Development & Optimisation : Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data … technical solutions. Communicate technical concepts effectively to both technical and non-technical audiences. Participate in code reviews and knowledge sharing sessions. Automation & DevOps: Implement automation for data pipeline deployments and other data engineering tasks. Work with DevOps teams to implement and Build CI/CD pipelines, for environmental deployments. Promote and implement DevOps best practices. … Azure Blob Storage, and Azure SQL Database. Experience working with large datasets and complex data pipelines. Experience with data architecture design and data pipeline optimization. Proven expertise with Databricks, including hands-on implementation experience and certifications. Experience with SQL and NoSQL databases. Experience with data quality and data governance More ❯

Data Platform, managing the platform, and driving its further development. This role requires proficiency in data engineering, data modelling, and data pipeline orchestration, with a proven track record of working with Azure-specific data platform tools and technologies. Strategic and forward-thinking capabilities, as well as experience in applying … and mentorship to the Data & Analytics team. Oversee the team’s Agile processes including sprints, ask prioritisation and issue resolution Define best practices for data pipeline design, development, and maintenance. Coordinate with the solution architect to ensure platform architecture meets current and future requirements. Act as a point of escalation for complex data … clean, usable datasets for reporting and analytics. Platform Administration and Optimization: Monitor and optimize Azure Synapse Analytics performance, ensuring cost efficiency and resource utilization. Implement monitoring and alerting for pipeline performance, failures, and resource consumption. Suggest and implement automation for repetitive tasks using tools like Azure Logic Apps, Power Automate, or custom scripting. Collaborate with the Data More ❯

that solves real-world problems for large blue-chip companies and governments worldwide. We're pioneers of meaningful AI : our solutions go far beyond chatbots. We are using data and AI to solve the world's biggest issues in telecommunications, sustainable water management, energy, healthcare, climate change, smart cities, and other areas that have a real impact on … re underpinned by over 300 engineers and scientists working across Africa, Europe, the UK and the US. ABOUT THE ROLE Sand Technologies focuses on cutting-edge cloud-based data projects, leveraging tools such as Databricks, DBT, Docker, Python, SQL, and PySpark to name a few. We work across a variety of data architectures such as Data … modelling, ETL (Extract, Transform, Load) processes, and big data technologies it becomes possible to develop robust and reliable data solutions. RESPONSIBILITIES Data Pipeline Development: Design, implement, and maintain scalable data pipelines for ingesting, processing, and transforming large volumes of data from various sources using tools such as databricks More ❯

Job Description Job Title: Data Architect Location: London - 3 days travel to office SC Cleared: Required Job Type: Full-Time Experience: 10+ years Job Summary : We are seeking a highly experienced and visionary Data Architect to lead the design and implementation of the data architecture for our cutting-edge Azure Databricks platform focused on … economic data. This platform is crucial for our Monetary Analysis, Forecasting, and Modelling efforts. The Data Architect will be responsible for defining the overall data strategy, data models, data governance framework, and data integration patterns. This role requires a deep understanding of data warehousing principles, big data technologies, cloud computing (specifically Azure), and a strong grasp of data analysis concepts within the economic domain. Key Experience: Extensive Data Architecture Knowledge: They possess a deep understanding of data architecture principles, including data modeling, data warehousing, data integration, and data governance. Databricks Expertise More ❯

Senior Data Engineer Salary: around £75k - plus a very attractive on top Location: London or Leeds - relaxed about hybrid working, if preferred About Us Sports forecasting company specializing in developing player level, play-by-play simulators that generate highly accurate, near-instant outcome projections for sporting events. Since its establishment, the company has focused on serving the emerging … grown into a globally recognized leader in sports modelling and analytics. Purpose of role: The purpose of this role is to design, build, and maintain scalable and efficient data transformation workflows that empower the business with actionable insights and impactful visualisations. This role bridges the gap between data engineering and business intelligence by owning the transformation … layer and enabling clear, trusted, and timely analytics across the organisation. The successful candidate will have a strong grasp of modern data modelling practices, analytics tooling, and interactive dashboard development in Power BI and Plotly/Dash. Key responsibilities: Designing and maintaining robust data transformation pipelines (ELT) using SQL, Apache Airflow, or similar tools. Building and More ❯

Senior Data Engineer Salary: around £75k - plus a very attractive on top Location: London or Leeds - relaxed about hybrid working, if preferred About Us Sports forecasting company specializing in developing player level, play-by-play simulators that generate highly accurate, near-instant outcome projections for sporting events. Since its establishment, the company has focused on serving the emerging … grown into a globally recognized leader in sports modelling and analytics. Purpose of role: The purpose of this role is to design, build, and maintain scalable and efficient data transformation workflows that empower the business with actionable insights and impactful visualisations. This role bridges the gap between data engineering and business intelligence by owning the transformation … layer and enabling clear, trusted, and timely analytics across the organisation. The successful candidate will have a strong grasp of modern data modelling practices, analytics tooling, and interactive dashboard development in Power BI and Plotly/Dash. Key responsibilities: Designing and maintaining robust data transformation pipelines (ELT) using SQL, Apache Airflow, or similar tools. Building and More ❯

over 10,000 people. Its big-scale stuff and we’re still growing. Job Purpose With a big investment into Databricks, and with a large amount of interesting data, this is the chance for you to come and be part of an exciting transformation in the way we store, analyse and use data in a fast … paced organisation. You will join as a Senior Data Platform Engineer providing technical leadership to the Data Engineering team. You will work closely with our Data Scientists and business stakeholders to ensure value is delivered through our solutions. Job Accountabilities ·Develop robust, scalable data pipelines to serve the easyJet analyst and data … time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. ·Understanding of the challenges faced in the design and development of a streaming data pipeline and the different options for processing unbounded data (pubsub, message queues, event streaming etc) ·Understanding of the most commonly used Data Science and Machine Learning More ❯

to technology, delivering bespoke software solutions and expert advice. Our clients are increasingly looking to us to help them make the best use of their data. In building data platforms and pipelines, our data engineers create the foundation for diverse data & analytics solutions, including data science and AI. They build data lakes and warehouses, create the processes to extract or access operational data, and transform siloed datasets into integrated data models that allow insight into business performance and problems or training of ML models. These are hands-on, client-facing roles, with openings at senior or lead level to suit your experience. You may be … or solving tough engineering challenges. You'd also be expected to spend some time on-site with clients in the London area on an ad-hoc basis. Our data engineers combine a strong software engineering approach with solid data fundamentals and experience with modern tools. We're technology agnostic, and we're open minded when it More ❯

Join to apply for the Senior Data Engineer (Remote) role at Circana Join to apply for the Senior Data Engineer (Remote) role at Circana At Circana, we are fueled by our passion for continuous learning and growth, we seek and share feedback freely, and we celebrate victories both big and small in an environment that is … entirely on what current employees say about their experience working at Circana. Learn more at www.circana.com. What will you be doing? We are seeking a skilled and motivated Data Engineer to join a growing global team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure … scalability. If you have a passion for data engineering and a desire to make a significant impact, we encourage you to apply! Job Responsibilities ETL/ELT Pipeline Development: Design, develop, and optimize efficient and scalable ETL/ELT pipelines using Python, PySpark, and Apache Airflow. Implement batch and real-time data processing solutions using More ❯

Senior Data Engineer – London - Hybrid Mars is a family-owned business with more than $35 billion in global sales. We produce some of the world’s best-loved brands: M&M’s®, SNICKERS®, TWIX®, MILKY WAY®, DOVE®, PEDIGREE®, ROYAL CANIN®, WHISKAS®, EXTRA®, ORBIT®, 5TM, SKITTLES®, BEN’S ORIGINAL®, and COCOAVIA®. Alongside our consumer brands, we proudly take … A capabilities and maintain them. Partnerships with other teams such as Supply, Manufacturing & R&D are also key to ensure process alignment and shared value creation. As a Data Engineer in the Commercial team, your key responsibilities are as follows: 1. Technical Proficiency: Collaborate in hands-on development using Python, PySpark, and other relevant technologies to create and … and Databricks. Proficiency in working with the cloud environment and various platforms, including Azure, SQL Server. NoSQL databases is good to have. Hands-on experience with data pipeline development, ETL processes, and big data technologies (e.g., Hadoop, Spark, Kafka). Experience with DataOps practices and tools, including CI/CD for data pipelines. More ❯

Senior Data Engineer – London - Hybrid Mars is a family-owned business with more than $35 billion in global sales. We produce some of the world’s best-loved brands: M&M’s®, SNICKERS®, TWIX®, MILKY WAY®, DOVE®, PEDIGREE®, ROYAL CANIN®, WHISKAS®, EXTRA®, ORBIT®, 5TM, SKITTLES®, BEN’S ORIGINAL®, and COCOAVIA®. Alongside our consumer brands, we proudly take … A capabilities and maintain them. Partnerships with other teams such as Supply, Manufacturing & R&D are also key to ensure process alignment and shared value creation. As a Data Engineer in the Commercial team, your key responsibilities are as follows: 1. Technical Proficiency: Collaborate in hands-on development using Python, PySpark, and other relevant technologies to create and … and Databricks. Proficiency in working with the cloud environment and various platforms, including Azure, SQL Server. NoSQL databases is good to have. Hands-on experience with data pipeline development, ETL processes, and big data technologies (e.g., Hadoop, Spark, Kafka). Experience with DataOps practices and tools, including CI/CD for data pipelines. More ❯

Employment Type:** Permanent **Salary:** Up to £62,000 **Location:** in the West of London **Work Arrangement:** In-office only, five days a week. **Job Title:** Data Architect/Data Engineer **Experience Required:** 6-7Years **Role Overview:** We are seeking a Senior Data Architect/Data Engineer to design and implement scalable … data solutions, including real-time and ETL/ELT data pipelines. Responsibilities include optimizing our data warehouse, creating Power BI reports, and developing AI-driven solutions using OpenAI APIs. **Key Responsibilities:** - Design and implement robust real-time data pipelines from diverse sources. - Maintain a high-performance data warehouse for reporting … and analytics. - Develop insightful Power BI dashboards and reports for stakeholders. - Use Azure Data Factory, Azure Functions, and Logic Apps for data integration. - Employ Python and SQL for data transformation and analysis. - Leverage OpenAI APIs for AI/ML solutions. - Collaborate with teams to align data strategy with business goals. - Ensure data More ❯

Employment Type:** Permanent **Salary:** Up to £62,000 **Location:** in the West of London **Work Arrangement:** In-office only, five days a week. **Job Title:** Data Architect/Data Engineer **Experience Required:** 6-7Years **Role Overview:** We are seeking a Senior Data Architect/Data Engineer to design and implement scalable … data solutions, including real-time and ETL/ELT data pipelines. Responsibilities include optimizing our data warehouse, creating Power BI reports, and developing AI-driven solutions using OpenAI APIs. **Key Responsibilities:** - Design and implement robust real-time data pipelines from diverse sources. - Maintain a high-performance data warehouse for reporting … and analytics. - Develop insightful Power BI dashboards and reports for stakeholders. - Use Azure Data Factory, Azure Functions, and Logic Apps for data integration. - Employ Python and SQL for data transformation and analysis. - Leverage OpenAI APIs for AI/ML solutions. - Collaborate with teams to align data strategy with business goals. - Ensure data More ❯

We are currently looking for a Senior Data Engineer to join us remotely in the UK. We plan to set up a hub in the UK in the following 12-18 months to enable our GTM strategy, and this would be our first hire in the region. You will work with our EMEA and US customers to help … them solve their data engineering, ML, ML Ops, and cloud migration puzzles. "As a Data Engineer at DATAPAO, you will have the unique opportunity to push the boundaries of technology and lead the way in how data transforms the world. We work with customers who want to innovate and change their industries. So it … GCP); You have a proven track record working with Databricks (PySpark, SQL, Delta Lake, Unity Catalog); You have extensive experience in ETL/ELT development and data pipeline orchestration (e.g., Databricks Workflows, DLT, Airflow, ADF, Glue, and Step Functions.); You’re proficient in SQL and Python , using them to transform and optimize data like a More ❯

Full-time | Hybrid | London | Up to £95,000 + Bonus & Excellent Benefits | Key Responsibilities: Design, develop, and implement advanced data pipelines and ETL/ELT workflows using cloud-native services such as AWS Glue, Lambda, S3, Redshift, and EMR. Act as a technical authority in cloud data engineering by mentoring colleagues and promoting best practices. Collaborate … cross-functionally with analysts, data scientists, and business stakeholders to translate data requirements into robust technical solutions. Establish and enforce standards around data quality, validation, lineage, security, and governance. Troubleshoot and resolve complex production issues and performance bottlenecks in data systems. Develop and maintain CI/CD pipelines for data workflows leveraging tools like Jenkins. Automate data ingestion, transformation, and loading processes to enhance operational efficiency. Work closely with DevOps and Engineering teams to ensure seamless deployment and maintenance of data infrastructure. Stay updated on emerging cloud computing and data engineering technologies to continuously improve team capabilities. Lead small-scale projects or More ❯

Location: Guernsey, Haywards Heath, Home Office (Remote) or Manchester Salary: £50,000 - £77,500 - depending on experience Department: Technology and Data We’re First Central Insurance & Technology Group (First Central for short), an innovative, market-leading insurance company. We protect the things customers love so they can get on with what matters to them in life. Data drives us. It fuels our outstanding distribution, finance, technology and legal services. Our underwriting skills are built on data expertise; it creates the insights we need to give the right cover to the right customers at the right price. But, it’s the people inside and outside our business that power us. They make us stand … career in a fast-paced and dynamic workplace? If so, today might just be your lucky day! We’re on the hunt for 2 passionate and skilled Senior Data Engineers to become key players in our cutting-edge technology and data teams. In this hands-on technical role, you’ll be responsible for building technical data More ❯

Data & AI Apps Engineer Department: Connectivity Employment Type: Permanent Location: Remote, UK Description The Data & AI Solutions Developer supports Nasstar's Data & AI Apps practice in designing, building and deploying modern data-driven and AI-powered software solutions on Microsoft technologies. The role focuses on rapid prototyping, proofs of concept, and production … over of AI solutions that leverage AI Foundry, Azure OpenAI, Microsoft Fabric, Copilot Studio, Power Platform and wider Azure services. Working under the guidance of senior engineers, the Data & AI Solutions Developer turns business and technical requirements into demonstrable value for customers, while continuously developing their own skills and certifications. The typical applications we build use existing closed … understanding of LLMs and other foundational models capabilities and limitations. Familiarity with Agile/Scrum ways of working and participation in sprint ceremonies Exposure to ETL/ELT pipelines, data modelling, or ML/AI workloads. Skills/Aptitude: Programming in Python, SQL, C#, and/or React (or similar); ability to write clean, maintainable code. Proficient with More ❯

to technology, delivering bespoke software solutions and expert advice. Our clients are increasingly looking to us to help them make the best use of their data. In building data platforms and pipelines, our data engineers create the foundation for diverse data & analytics solutions, including data science and AI. They build data lakes and warehouses, create the processes to extract or access operational data, and transform siloed datasets into integrated data models that allow insight into business performance and problems or training of ML models. These are hands-on, client-facing roles, with openings at senior or lead level to suit your experience. You may be … leading teams, setting technical direction, advising clients or solving tough engineering challenges. Our data engineers combine a strong software engineering approach with solid data fundamentals and experience with modern tools. We’re technology agnostic, and we’re open minded when it comes to your existing skillset. What are we looking for? Good experience with some of More ❯

Nadara Greater London, England, United Kingdom R0001200 Data Architect (Open) We’re Nadara. We work harmoniously with the power of nature and the communities around us to power lifetimes to come. We call our approach ‘living energy’. We don’t just produce renewable energy, we live it – recognising our relationship with the people touched by our business … different perspectives with respect. Discover more about our culture and approach here: https://nadara.com/living-the-company/Job Description Summary Nadara is hiring a Data Architect with expertise in cloud data platforms to design, evolve, and optimize our enterprise data platforms. This role will be responsible for developing company’s … cloud-based data platform environment and ensure its operations are optimized to performance and scalability standards. The Data Architect will also provide technical leadership in data, analytics, and AI, collaborating with the Data, Analytics, and AI team, business stakeholders, and third-party partners to operate, manage, and develop Nadara’s cloud computing More ❯

Job Description Department: Information Technology Job Summary We are seeking a highly skilled and motivated Data Scientist with a strong focus on machine learning (ML) and artificial intelligence (AI) to join our innovative team. The ideal candidate will excel in developing, deploying, and optimizing ML models and AI solutions, leveraging cutting-edge technologies like Azure Machine Learning, AutoML … to manage model lifecycles. Leverage cloud platforms like Azure, AWS, and GCP for scalable ML model deployment. Employ frameworks like TensorFlow, PyTorch, and scikit-learn for model development. Data Engineering and Preparation Oversee data ingestion, cleaning, transformation, and feature engineering processes to ensure high-quality datasets. Work with large datasets and implement scalable data … and maintain a centralized repository. Establish and follow best practices in model versioning, reproducibility, and model governance. Qualifications Education Advanced degree (Master’s or Ph.D.) in Computer Science, Data Science, Statistics, or a related field. Equivalent experience will be considered. Experience Minimum of 5 years in a data science or machine learning-focused role. Technical Skills More ❯

different format of this document, please get in touch with at [emailprotected] or call TCS London Office number 02031552100 with the subject line: “Application Support Request”. Role : Data Architect Job Type: Permanent (Hybrid) Location: London, United Kingdom Ready to utilize your skills in designing, creating, and managing data architecture? Join us as a Data … learning opportunities. • Be part of an exciting team where you will be challenged every day. • Build strong relationships with a diverse range of stakeholders. The Role As a Data Architect, you will be responsible for designing, creating, and managing data architecture. You will also ensure that data is efficiently and securely stored, organized, and … accessible across the enterprise. Build the foundation for databases, data warehouses, data lakes, and other data storage solutions, ensuring they meet both business and technical requirements. Key responsibilities: - Develop and design the data architecture framework for the organization. - Create models for databases, data warehouses, data lakes, and other More ❯