1 to 25 of 30 Data Pipeline Jobs in Berkshire

a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets. Key Responsibilities: Data Pipeline Development & Optimisation: Design, develop, and maintain robust and scalable data pipelines for ingesting, transforming, and loading data from various sources (e.g., APIs, databases, financial data … technical solutions. Communicate technical concepts effectively to both technical and non-technical audiences. Participate in code reviews and knowledge sharing sessions. Automation & DevOps: Implement automation for data pipeline deployments and other data engineering tasks. Work with DevOps teams to implement and Build CI/CD pipelines, for environmental deployments. Promote and implement DevOps best practices. … Azure Blob Storage, and Azure SQL Database. Experience working with large datasets and complex data pipelines. Experience with data architecture design and data pipeline optimization. Proven expertise with Databricks, including hands-on implementation experience and certifications. Experience with SQL and NoSQL databases. Experience with data quality and data governance More ❯

CloudXcel is a technology-driven company dedicated to providing innovative cloud solutions to businesses worldwide. Our mission is to empower organizations to harness the full potential of their data in the cloud, unlocking insights and driving transformation. With a commitment to cutting-edge technologies and a dynamic, inclusive work environment, we foster growth, creativity, and innovation. Overview: CloudXcel … is seeking a talented and highly motivated Cloud Data Engineer with expertise in Python to join our team. In this role, you will work closely with cross-functional teams to design, develop, and optimize cloud-based data pipelines and infrastructure. The ideal candidate is passionate about data engineering, proficient in Python, and eager to … solve complex challenges in a fast-paced, collaborative environment. Key Responsibilities: Design, develop, and maintain scalable cloud-based data pipelines to collect, process, and analyze large datasets. Build and optimize ETL workflows to ensure efficient data integration and processing. Collaborate with data scientists, analysts, and engineers to support data modeling, analytics, and More ❯

integrated teams work across disciplines, applying digital innovation and best-in-class project delivery. Role Overview: We are seeking a talented and proactive Power BI Specialist to support data-driven decision-making on some of the most complex infrastructure programs in the region. You will be responsible for developing insightful dashboards and reports that drive operational performance, track … billion-dollar infrastructure, energy, and transportation projects. Key Responsibilities: Develop and maintain interactive dashboards and visualisations using Power BI to support project teams, clients, and senior stakeholders. Integrate data from diverse systems including project management software (Primavera P6, MS Project), finance (SAP, Oracle), GIS, IoT, and Excel-based tools. Collaborate with multidisciplinary teams (engineering, project controls, digital delivery … and data governance) to understand reporting requirements and KPIs. Support real-time project monitoring and automated reporting solutions for cost, schedule, risk, safety, and sustainability metrics. Apply Power Query, DAX, and SQL for data modeling, transformation, and performance optimisation. Ensure compliance with data governance, security protocols, and company standards for digital delivery. Support the More ❯

integrated teams work across disciplines, applying digital innovation and best-in-class project delivery. Role Overview: We are seeking a talented and proactive Power BI Specialist to support data-driven decision-making on some of the most complex infrastructure programs in the region. You will be responsible for developing insightful dashboards and reports that drive operational performance, track … billion-dollar infrastructure, energy, and transportation projects. Key Responsibilities: Develop and maintain interactive dashboards and visualisations using Power BI to support project teams, clients, and senior stakeholders. Integrate data from diverse systems including project management software (Primavera P6, MS Project), finance (SAP, Oracle), GIS, IoT, and Excel-based tools. Collaborate with multidisciplinary teams (engineering, project controls, digital delivery … and data governance) to understand reporting requirements and KPIs. Support real-time project monitoring and automated reporting solutions for cost, schedule, risk, safety, and sustainability metrics. Apply Power Query, DAX, and SQL for data modeling, transformation, and performance optimisation. Ensure compliance with data governance, security protocols, and company standards for digital delivery. Support the More ❯

Join a Leading Growth business on its transformative journey to become the UK's most recommended businesses in their industry Lead Data Engineer Salary: £70-90k Location: Reading – Hybrid Benefits: Competitive Role Overview: As Lead Data Engineer , you will be part of the Corporate Data team, reporting directly to the Senior Leadership team. … You will lead a small team of data engineers, overseeing the design, build, and maintenance of the infrastructure and systems that collect, store, and process corporate data for reporting and analysis. Key Responsibilities: • Lead and manage a team of data engineers. • Design, build, and maintain data infrastructure and systems. • Ensure effective use … of Microsoft Fabric and Synapse analytics platform. • Collaborate with various stakeholders, including Data and Systems owners, Data Analysts, Data Architects, Infrastructure Architects, and external suppliers. Experience required • Dynamic and hands-on leader who excels in managing and coaching a team. • Thrives in a collaborative environment and adapts to changing priorities with ease. • Excellent communication More ❯

for our “nothing is impossible” ethos, we prioritize trust, transparency, and innovation in everything we do. About the Role: We are looking for a Machine Learning Engineer with Data Engineering expertise to help scale our platform. In this hybrid role, you’ll design data pipelines, develop ML models, and work across data and AI … machine learning models for production. • Optimize model performance based on key metrics for scalability, reliability, and real-world impact. • Build and maintain end-to-end ML pipelines, including data preprocessing, model training, deployment, and monitoring. • Work closely with cross-functional teams to integrate ML models into our SaaS platform for PAI and OSINT investigations. Data Engineering … Develop, maintain, and optimize scalable data pipelines for ingesting, processing, and storing large volumes of data. • Ensure data quality, consistency, and availability to support ML workflows. • Work with ELT processes and implement Medallion (Bronze/Silver/Gold) architecture to structure and optimize data transformation. • Align data infrastructure with business needs and More ❯

commodities trading houses, leveraging sophisticated technology to trade a broad range of energy, metals, and agricultural commodities globally. Our success is driven by cutting-edge analytics, high-frequency data processing, and a commitment to innovation. We are now looking for a Senior Python Engineer to join our dynamic team in London and drive our next wave of technological … Role Overview: As a Senior Python Engineer, you will be at the heart of our technology platform, building robust systems that enhance our trading strategies, risk management, and data analysis. You will collaborate closely with quants, data scientists, and traders, providing Python-based solutions that directly impact the profitability of the business. This is an exciting … can push the boundaries of technology in financial markets. Key Responsibilities: Design, develop, and maintain highly efficient Python-based applications for commodities trading, risk management, and real-time data analysis. Build scalable, high-performance data pipelines to process large datasets from multiple sources, including real-time market data and fundamental data. Implement robust APIs More ❯

Company: My client are a fast-growing fintech transforming how people manage and move money. As a Data Engineer, you’ll play a pivotal role in building and scaling the data infrastructure that powers everything from real-time fraud detection to personalized financial insights. Key Responsibilities: Design, build, and maintain scalable data pipelines (batch … streaming) Work with Data Scientists and Analysts to enable fast, reliable access to data Optimize our data architecture for performance, reliability, and cost Collaborate with engineers to integrate data best practices across teams Champion data quality, governance, and documentation Key Requirements: Strong experience with Python, SQL, and modern ETL tools … e.g., Airflow, dbt) Solid grasp of cloud platforms (AWS/GCP/Azure) and data warehouses (e.g., BigQuery, Snowflake) Familiarity with streaming technologies (Kafka, Kinesis, etc.) Passion for clean, maintainable code and robust data systems Previous experience in a fintech or regulated environment is a plus Benefits: Join a collaborative, mission-driven team at the cutting More ❯

recognition and inclusivity and offer a wide range of benefits to support staff wellbeing. Your Future Starts Here PURPOSE OF JOB: We’re looking for an experienced Azure Data & AI Engineer with a strong focus on advanced analytics, machine learning, and applied AI - particularly Generative AI. This role will suit a technically capable professional who combines real-world … Vault), and telemetry via Application Insights or Log Analytics. KEY RESPONSIBILITIES: Have 5+ years of technical consulting or a similar chronology of demonstrable success in coding and deploying Data and AI models onto Azure environments. At least one year using Azure Fabric Create scalable and reliable data pipelines for data processing, transformation, and storage. … Establish and enforce data governance policies and be ‘hands-on’ in exploratory data analysis to identify and measure quality gaps that are a block to Data and Ai implementation. Implement robust security measures to protect sensitive data within Azure environments. Integrate AI and machine learning models into data pipelines and More ❯

recognition and inclusivity and offer a wide range of benefits to support staff wellbeing. Your Future Starts Here PURPOSE OF JOB: We’re looking for an experienced Azure Data & AI Engineer with a strong focus on advanced analytics, machine learning, and applied AI - particularly Generative AI. This role will suit a technically capable professional who combines real-world … Vault), and telemetry via Application Insights or Log Analytics. KEY RESPONSIBILITIES: Have 5+ years of technical consulting or a similar chronology of demonstrable success in coding and deploying Data and AI models onto Azure environments. At least one year using Azure Fabric Create scalable and reliable data pipelines for data processing, transformation, and storage. … Establish and enforce data governance policies and be ‘hands-on’ in exploratory data analysis to identify and measure quality gaps that are a block to Data and Ai implementation. Implement robust security measures to protect sensitive data within Azure environments. Integrate AI and machine learning models into data pipelines and More ❯

Job Title: Contract Data Architect Location: Reading (Hybrid or On-site as required) Contract Length: 3 Months Sector: Financial Services Overview: We are seeking an experienced Data Architect to support a leading financial services organisation based in Reading. This is a 3-month contract role focused on designing and implementing scalable, secure, and compliant data solutions in a highly regulated environment. Key Responsibilities: Design and deliver end-to-end data architecture solutions to support business intelligence, reporting, and regulatory needs Lead the integration and optimisation of large-scale data platforms using Azure Synapse and Databricks Build and maintain robust data pipelines using Python (PySpark) and SQL Collaborate … with data engineers, analysts, and stakeholders to ensure data quality, governance, and security Ensure all solutions adhere to financial regulations and internal compliance standards Key Skills & Experience: Proven experience as a Data Architect within the financial services sector Hands-on expertise with Azure Synapse Analytics and Databricks Strong programming and data engineering More ❯

use cutting-edge technology to link our clients' systems, departments and sites. We provide an open technology platform that's shaping the industry for the future. We use data to help clients become more efficient, increase profitability and give more customers an amazing experience. Want to be part of it? Position Summary We are hiring a AI & Data Science Lead to head our AI Centre of Excellence. This role consolidates responsibilities across Generative AI (GenAI) , traditional Machine Learning (ML) , and early-stage Data Science . You will shape and deliver the technical strategy for our AI initiatives, while also acting as a hands-on engineer, mentor, and leader. You will be responsible for delivering … high-impact GenAI use cases, establishing core AI platform capabilities, supporting a Machine Learning Engineer on classical ML projects, and building the foundation for a future Data Science function. You will also define LLMOps and MLOps practices and ensure the infrastructure is in place to support long-term AI success across the business. Key Responsibilities: Strategy & Leadership Collaborate More ❯

security architecture for AI/ML systems, CI/CD and data pipelines. * Participate in enterprise-wide architecture initiatives for AI/ML. Understand the workflow and pipeline architectures of ML and deep learning workloads. * Conduct security risk assessments and threat modelling for AI/ML and other business projects performed thorough design reviews and security assessments … testing of AI/ML models, LLM and development of trustworthiness, safety, fairness, bias and inappropriateness detection, as well as adversarial test specification, development, and execution. * Collaborate with data scientists, engineers, and other stakeholders to embed security controls and measures into AI/ML applications, models, algorithms, and data pipelines. * Define and enforce robust access controls … authentication mechanisms, and encryption standards to safeguard AI/ML systems, training data, provenance, and any sensitive data. * Stay abreast of emerging AI security threats, trends, and technologies, evaluating their impact and proactively adapting security measures to mitigate potential risks. * Provide technical guidance and support to cross-functional teams during the design, development, and deployment of AI/ More ❯

with support Performance Bonus About the role As a mid-to-senior level Machine Learning Engineer, you will create ML models for deployment in the cloud and support Data Engineering, DataOps, and DevOps teams. You will work with the latest technology on highly visible, large-scale projects. What you'll bring to the role Strong grounding in ML … cloud and working with Azure (Azure DevOps, Azure ML, Azure SQL, ADF, etc.) Ability to work with databases such as SQL Server, NoSQL, etc. Experience with ETL processes, data pipelines, and DevOps practices This role is hybrid, requiring occasional presence in one of their UK offices in Scotland or England to maintain company culture through social meetups. You More ❯

A leading Financial Services client in the City of London is now seeking an experienced Data Engineering Manager to join on a permanent basis. This role is offering a base of £85,000 + a strong benefits package and flexible working. The ideal Data Engineering Manager will come from a data engineering background and … have strong knowledge in SQL, Snowflake, Microsoft Azure, Azure Data Factory and Azure DevOps. The Engineering Manager will design, improve and maintain robust data pipelines within data architecture. To be considered for this role you will need the following: Experience designing, improving and maintaining robust data pipelines Strong SQL programming skills. Knowledge … of other programming languages such as Python, C++ and Java beneficial Possesses a strong understanding of Snowflake - beneficial Experience managing small teams of Data Engineers Strong experience working in a cloud environment and knowledge in the following very beneficial: Microsoft Azure, Azure Data Factory and Azure DevOps Experience working in fast-paced Agile environments Creativity and More ❯

A leading Financial Services client in the City of London is now seeking an experienced Data Engineering Manager to join on a permanent basis. This role is offering a base of £85,000 + a strong benefits package and flexible working. The ideal Data Engineering Manager will come from a data engineering background and … have strong knowledge in SQL, Snowflake, Microsoft Azure, Azure Data Factory and Azure DevOps. The Engineering Manager will design, improve and maintain robust data pipelines within data architecture. To be considered for this role you will need the following: Experience designing, improving and maintaining robust data pipelines Strong SQL programming skills. Knowledge … of other programming languages such as Python, C++ and Java beneficial Possesses a strong understanding of Snowflake - beneficial Experience managing small teams of Data Engineers Strong experience working in a cloud environment and knowledge in the following very beneficial: Microsoft Azure, Azure Data Factory and Azure DevOps Experience working in fast-paced Agile environments Creativity and More ❯

Job Title: Data Lead Salary: up to ~£110k + very generous equity (of the same value as your salary!) Benefits include: unlimited AL, 2 company trips/year, contribution towards your own holiday Location: Shoreditch (2 office days/week) Industry: FinTech/Payments Company overview Having raised over £7m at seed stage, this FinTech company is shaking … largest players in payments, this company is causing the biggest change to how businesses access card payments in over two decades. Role overview As the company's first data hire, you’ll be tasked with designing, building and owning their data stack. You’ll also be responsible for applying your expertise in quantitative analysis to understand … how their users behave. Your insights will feed into data products that you’ll own, as well as into the wider product strategy. Reporting directly to the CTO/founder and working alongside the founding employees, you'll have a huge impact on the business from day 1 - you'll get to experiment and built innovative products at More ❯

Lead Backend Engineers (Platform Experience & Data Warehousing teams) Location: UK, Ireland, Poland, Lithuania, Spain, Portugal, Finland numi are proud to have partnered with a front runner in data integration technology providing fantastic experience for their clients (who are some of the biggest names in the space)! We are looking for experienced Lead Backend Engineers (PHP) to … join either Platform Experience or Data Warehousing teams. The Platform Experience team focuses on optimising end-to-end platform journeys and building impactful customer experiences. They are currently developing a data visualisation tool (dashboards) and build features primarily from scratch. The Data Warehousing team is responsible for data storage and transfer solutions … helping users streamline data transfers to data warehouses/datalakes and improving internal storage. This involves scaling, refactoring, and enhancing a complex data transfer system handling large data volumes. As a Lead Backend Engineer, you will lead the development of new features, drive high code quality, provide technical guidance, and collaborate with More ❯

Lead Backend Engineers (Platform Experience & Data Warehousing teams) Location: UK, Ireland, Poland, Lithuania, Spain, Portugal, Finland numi are proud to have partnered with a front runner in data integration technology providing fantastic experience for their clients (who are some of the biggest names in the space)! We are looking for experienced Lead Backend Engineers (PHP) to … join either Platform Experience or Data Warehousing teams. The Platform Experience team focuses on optimising end-to-end platform journeys and building impactful customer experiences. They are currently developing a data visualisation tool (dashboards) and build features primarily from scratch. The Data Warehousing team is responsible for data storage and transfer solutions … helping users streamline data transfers to data warehouses/datalakes and improving internal storage. This involves scaling, refactoring, and enhancing a complex data transfer system handling large data volumes. As a Lead Backend Engineer, you will lead the development of new features, drive high code quality, provide technical guidance, and collaborate with More ❯

Job Description: Are you passionate about Data and Analytics (D&A) and excited about how it can transform enterprise operations? Do you possess strategic vision, technical expertise, and leadership skills to drive data-driven solutions? If so, you might be the ideal candidate for the Data and Analytics role within Mars's Global Pet … achieve growth, profitability, and customer centricity. This role requires demonstrating thought leadership, technical proficiency, and the ability to navigate complex challenges while leading a team of top-tier data and analytics professionals. The role involves transitioning to a product-based model to develop digital capabilities. Responsibilities include developing and maintaining robust data pipelines and storage solutions … to support analytics and machine learning initiatives. You will report to the Director of Data Engineering Solutions and collaborate globally with engineering teams across core products. Key Responsibilities: Technical Leadership: Provide leadership to data and DevOps engineers. Collaborate on designing and evolving scalable data platforms. Promote best practices and foster a high-performance culture. More ❯

Head of Data Engineering Description of Role and Client: The Head of Data Engineering will lead the technical development and operational management of data engineering solutions for a prominent insurance organization. The client requires a data engineering expert capable of managing complex environments and collaborating with senior technical teams. This role falls … inside IR35. Responsibilities: Design, develop, and manage data pipelines, warehouses, and data lakes. Optimize and maintain enterprise data engineering platforms. Lead strategic management of data engineering solutions across cloud and hybrid environments. Collaborate closely with Data Architects, CIO teams, and senior stakeholders. Continuously enhance the efficiency, reliability, and scalability of … data engineering landscapes. Candidate Requirements/Profile: Extensive data engineering experience within insurance sectors. Proficiency with AWS, Azure, GCP, and hybrid data environments. Relevant certifications in cloud platforms or big data technologies. Demonstrated ability to strategically own and enhance enterprise data platforms. Contract duration: Initial 6-month term, with likely More ❯

customer segmentation, proidling and personalisation projects Help shape the data roadmap from early foundations to advanced maturity. What you'll bring: Strong SQL and data pipeline understanding (you will work with data engineering teams rather than being hands on coding pipelines). Proven experience building dashboards and defining KPIs. Comfortable with data More ❯

a Principal Python Engineer to act as a quantitative overlay on global equity trading operations. The engineering team's goal is to continually improve the entire ecosystem-spanning data, research, trading, and post-trade analytics. As a Principal Engineer, you will own end-to-end delivery of new features and projects, engaging in everything from trading framework design … closely with quants, traders, and other engineering teams to build a robust and scalable CRB platform. Responsibilities: Platform Development: Enhance and maintain the CRB ecosystem, including research tools, data pipelines, trading frameworks, back-testing infrastructure, and post-trade analysis systems. End-to-End Project Ownership: Collaborate with stakeholders to define requirements, design solutions, implement code, and oversee production … and other essential tooling. Performance Optimization: Identify and fix performance bottlenecks in multi-threaded systems, ensuring high availability and low latency. Technical Exploration: Investigate emerging technologies-particularly in data analysis and machine learning-and adapt them to evolving platform needs. Ideal Candidate Background: Experience: 7+ years of professional software development with exposure to capital markets or trading (equities More ❯

functionality, reliability, security. Tools: Postman, SoapUI, RestAssured, Swagger. Mobile Testing Focus on mobile app quality across Android/iOS devices. Manual and automation testing using Appium, XCUITest, TestFairy. Data/ETL Testing Ensures accurate data flow, transformation, and validation in data pipelines. Works with SQL, Informatica, Talend, SSIS, Snowflake. AI/ML Model Testing More ❯

functionality, reliability, security. Tools: Postman, SoapUI, RestAssured, Swagger. Mobile Testing Focus on mobile app quality across Android/iOS devices. Manual and automation testing using Appium, XCUITest, TestFairy. Data/ETL Testing Ensures accurate data flow, transformation, and validation in data pipelines. Works with SQL, Informatica, Talend, SSIS, Snowflake. AI/ML Model Testing More ❯