1 to 25 of 259 Data Pipeline Jobs in London

Data Visualisation Engineer Location: UK Salary: £41,000 per annum We are Forward, the social enterprise that empowers people to break the cycle of crime or addiction to move forward with their lives. For more than 25 years we have been working with people to build positive and … make positive changes and build productive lives with a job, family, friends and a sense of community. What you will be doing The Data Visualisation Engineer (DVE) is responsible for designing, developing, and maintaining data visualisation systems and dashboards. Using a variety of tools and technologies … to create interactive and visually appealing representations of data that enable users to quickly and easily understand complex data sets. Services are designed to deliver a range of interventions, programmes, workshops, advice and guidance for our clients. As a member of the Data & Performance more »

seizing energy investment opportunities. Job Summary The job holder will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection for cross functional teams. The Data Engineer will support our software … developers, database architects, and Trading teams with data initiatives that will ensure optimal data delivery architecture is consistent throughout ongoing projects. This role requires the jobholder to be self-directed and comfortable supporting the data needs of multiple teams, systems and products. They will … architecture to support our next generation of products and data initiatives. Key Responsibilities: Create and maintain optimal data pipeline architecture Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key more »

Senior Data Engineer About us HENI is a technology company pioneering art markets and information. We work with world-leading artists and estates across various sectors, including printmaking, book publishing, NFTs, digital content, and art research – all underpinned by cutting-edge research in analytics, AI, and machine learning. … Our team comprises of talented web developers, data engineers, data scientists, and infrastructure specialists who thrive on solving complex problems. We enjoy developing machine learning tools and embracing the latest in infrastructure and coding methodologies. We maintain high standards of technical quality via thorough testing and … best of open-source data engineering and scientific Python toolset. Our tech stack includes Airbyte for data ingestion, Prefect for pipeline orchestration, AWS Glue for managed ETL, along with Pandas and PySpark for pipeline logic implementation. We utilize Delta Lake and PostgreSQL for data more »

Duration - 12 months We are seeking a Data Engineer who will be responsible for processing and managing our high-quality multimedia datasets for use by our researchers and engineers. The ideal candidate will have a strong background … in data engineering, experience in data processing and management, and proficiency in working with developers to optimise data pipeline tooling. Responsibilities: Data processing: Using online workflows and command line tools/scripts to clean and transform our multimedia datasets to make … sharing other relevant information. This requires collaboration with multiple internal teams and working with spreadsheets, documents, internal wikis, and similar tools. Data pipeline: Support improvements to our data pipeline tooling by working closely with developers to provide feedback and make modifications to optimise data more »

This role falls outside IR35*** James Adams is currently looking for an experienced Lead Data Engineer. This is a 6-month initial contract working hybrid remotely in London. Seeking a Talented Lead Data Engineer My client is looking for an experienced and skilled Lead Data Engineer to join their growing team. You'll play a vital role in designing, building, and maintaining their data infrastructure, supporting their mission of making industry data more transparent and accessible. Responsibilities: Lead the design, development, maintenance, testing, and optimization of a scalable and … documentation for the data platform, including data processes, schemas, and dictionaries. Identify and implement improvements to enhance data pipeline efficiency and reliability. Manage cloud-based data services, balancing performance and cost across storage and compute capabilities. Mentor and guide junior data more »

As the Data Architect, you'll lead data architecture initiatives, establishing standards and practices essential for effective data management. Your focus will be on designing and implementing robust data and information systems, ensuring they meet stringent criteria for modelling, flow, lineage, quality … and security. Working closely with stakeholders and data governance teams, you'll drive adherence to established standards, facilitating data-driven decision-making processes across the organisation. This role encompasses: Developing and maintaining a comprehensive Data Architecture Framework and maturity roadmap. Enforcing best data practices, setting standards, and ensuring compliance. Creating data models and overseeing the implementation of changes to address evolving business needs. Providing leadership in evaluating and implementing master data management and metadata management solutions. Collaborating with IT operations and delivery teams, producing essential data more »

Data Engineer Location: London, 3 days per week in the office Salary range: £50,000-£70,000 + benefits Purpose: Design, build, and maintain scalable data architectures, including pipelines and cloud-based data warehouses. Tech: Python (NumPy, Pandas), SQL, ETL, Cloud (AWS, Azure or … consultancy who solely specialise in the Maritime trading industry. You will work onsite with 1 specific client to design, build and maintain scalable data architectures working with key stakeholders in their business to shape their data-driven strategies. The Company: They have a long history providing … ETL Processes: Deep expertise in developing and optimising ETL pipelines. Version Control: Experience with Git for code collaboration and change tracking. Data Pipeline Tools: Proficiency with tools such as Apache Airflow. Cloud Platforms: Familiarity with AWS, Azure, Snowflake, and GCP. Visualisation: Tableau or PowerBI Delivery Tools: Familiarity more »

We Want We are looking for a Staff Backend Engineer to join our Insights Back-End team to help build out data pipeline, data store and API layer of Bazaarvoice data that will power our various client-facing dashboards, raw data reporting, product performance reporting. The ideal candidate has built and operated systems that aggregate and share data sets to support new product offerings and is well experienced in evaluating the right technologies to build those data sets. Success in the role is underpinned by … a strong understanding of the business cases of the product portfolio and building relevant data sets, API layer and access methodologies that support the portfolio goals. What You’ll Be Doing Own, design, implement and support of major components of the unified back end data. Define how data more »

SENIOR DATA SCIENTIST (NLP & LLM) Up to £85,000 LONDON Ready to elevate your career to new heights? Join a team of experienced Data Scientists and Data Engineers to create impactful LLM's models and work with cutting-edge NLP technologies. The Role: Work … performance, and make informed solution selections. Conduct comprehensive work spanning the entire data lifecycle, including data exploration, data pipeline development, insights and feature extraction, machine learning model creation, and deployment. Collaborate on consulting-style pilot projects aimed at helping clients gain a deep … understanding of their data and identifying problems that can be addressed through the platform. Collaborate closely with data scientists and data engineers to construct and implement machine learning models. Take an active role in leading team meetings and client workshops. Assist in establishing software more »

developer 6 months contract – Competitive market rates – Inside IR35 - London – 3 days on-site Key Skills: Python, Power BI, SQL, Experience with Big Data pipelines, DAX, Excel, Financial services background The Role: A global trading organisation is looking a Python/Power BI to join their Data … be working with the business helping to build Apps using Python and Power BI for the output dashboards. Key Responsibilities: Develop and construct data pipelines to facilitate efficient extraction, transformation, and loading of data from diverse sources. Create and implement Business Information Access portals, comprising dashboards … continued deployment and enhancements. Innovate existing solutions and forecast future demands to meet evolving business needs. Develop analytics tools leveraging the data pipeline to offer actionable insights across various business functions, thereby enhancing key performance metrics. Emphasize automation and adopt an early feedback approach to streamline processes. more »

Stott and May has partnered with a global healthcare organisation within the private sector, to source a senior data engineer to join their growing tech team. Hybrid (3/4 days per month in London or Manchester) Salary: Circa £80k p/annum Key Responsibilities: Lead the consolidation … and migration of legacy data to our new cloud-based data platform. Design and implement master and transactional data models to ensure data integrity and consistency. Develop ETL pipelines and cloud data storage solutions that meet business requirements and non … functional needs. Collaborate closely with data analysts, visualization engineers, and project managers to deliver data solutions. Surface data for analytical consumption and prepare data for ML/predictive and prescriptive models. What we are looking for: Proficiency with Databricks and building data more »

Senior Data Engineer - Azure Paying up to 70K Hybrid: 2 days onsite About the Role My food retail business client is looking for a senior Data Engineer to join the team. My client operate on a global scale, but their Data and Analytics function … career progression opportunities as part of our technology transformation journey. Paying up to £70K Hybrid (2 days onsite) Main Duties Design and deliver data pipelines. Build and manage Azure Data Factory data flows. Ingest data into an Azure Data Lake. … Design data models for large datasets, including creating dimensional multivariate models. Utilize non-relational data storage solutions such as MongoDB. Work with SQL Server Integration Services (SSIS) and other ETL tools. Develop solutions with SSIS, SSRS, T-SQL (On-prem/Cloud). Manage NoSQL environments more »

My leading Energy and Commodities conglomerate are currently seeking a skilled individual with experience in Python-based data pipeline, data lake house development and an appreciation for Object orientated languages such as C# or Java, using leading orchestration tools. In this role, you'll lead … the development of robust, fully tested data pipelines in Python using cutting-edge platforms like Dagster/Airflow and apply your expertise in real-time data streaming solutions using Kafka. You'll play a key role in expanding their on prem data lake infrastructure … to help enhance support for new lines of businesses they have including Systematic trading, Trading and Shipping commodities, ensuring strong data lineage. The successful candidate will have: A strong background in data engineering using Python Strong understanding of Test Driven Development Experience building Star Schemer Databases more »

Job Title: Data Operations Analyst Company: Global Asset Manager Location: City of London, 3 days per week in office Salary: £85,000 per annum + Benefits About the Company: Join a prestigious global asset manager with over $200 billion in assets under management. Our firm is renowned for … excellence. Based in the heart of London's financial district, you'll be part of a dynamic team shaping the future of financial data operations. Role Overview: As a Data Operations Analyst, you will play a pivotal role in implementing a cutting-edge global platform aimed … at centralizing data management processes. Leveraging your expertise in R, Python, SQL, and Enterprise Data Management (EDM), you'll contribute to streamlining data workflows and optimizing data quality across the organization. Key Responsibilities: Collaborate with cross-functional teams to understand data more »

Job Description Lead Data Engineer (Java/Python/AWS/Glue/Dremio). Investment Management. £160,000 - £170,000 + 25% Bonus Target and Benefits. Hybrid 2 Days a week in St Pauls Office. My client is a top tier multi asset investment management house based … in Central London. The Data Office team is playing a key role in helping build the future of financial services, working together with business partners to create client experiences that are changing the way people invest. You will work with smart, talented people across our business. We will … platform’s architectural design Development and testing of the integration, modelling, data persistence and querying tools and analytical systems Data pipeline implementation, maintenance and testing Metadata management processes and tools Monitoring the overall performance and stability of the pipelines Implementing data curation, metadata more »

Job Description Senior Data Engineer. Java or Python and AWS. Investment Management. 12 month rolling Fixed Term Contract. £135,000 - £145,000 + 15% Guaranteed Bonus paid monthly + Benefits including health, holidays and 13% pension. The Data Office team at my client is playing a … view to the table—because that is how they help their clients succeed. You will be an individual contributor on the Analytics and Data Platform engineering team responsible for architecting, building, testing, and maintaining the data platform. You will be responsible for: Contributing to the … platform’s architectural design Development and testing of the integration, modelling, data persistence and querying tools and analytical systems Data pipeline implementation, maintenance and testing Metadata management processes and tools Monitoring the overall performance and stability of the pipelines Implementing data curation, metadata more »

since early 2022. FullCircl was formed in 2021 following the merger of Artesian Solutions and DueDil. In September 2023 FullCircl acquired W2 Global Data Solutions. FullCircl is backed by top-tier investors which include Octopus Investments, Notion Capital, Augmentum Fintech and notable angel investors which include Dr Steve … competitive compensation and benefits and believe employee ownership is a key part of cultivating the mindset we want from everyone. The Role Lead Data Engineer We realise at FullCircl that it takes a dedicated team of brilliant people to continually innovate, push boundaries and exceed expectations. We pride … amazing things. Still led by our three original founders, we’re constantly on the lookout for rising stars. We are seeking a lead data engineer to join our team and to work on a variety of assignments alongside our product and commercial management teams. The role will require more »

Job DescriptionSenior Data Engineer (Inside IR35) - £700 to 825 per dayLocation: Remote/Hybrid (London)Rate: £825 per dayContract: Inside IR35Are you an experienced Data Engineer with a strong background in banking? Our client are seeking Senior Data Engineers to contribute to innovative data projects within a leading financial institution.Key Responsibilities:Data Engineering: Design and implement data models, ETL processes, and data warehousing solutions.Programming: Utilize Python, Java, Scala, or GoLang to build and optimize data pipelines.Distributed Processing: Work with Hadoop, Spark, and other platforms … for large-scale data processing.Real-Time Data Streaming: Develop and manage pipelines using CDC, Kafka, and Apache Spark.Database Management: Handle SQL databases like Oracle, MySQL, or PostgreSQL.Data Governance: Ensure data quality, security, and compliance with best practices.Ideal Candidate:Significant experience in data more »

Job Title: Data Engineer Job Type: Permanent Location: London or Fully remote Salary: £50,000-£70,000 (Dependant on experience) + Bonus + Benefits Skills: Python, SQL, Posgress SQL, Snowflake, Powershell Roc Search`s client provides real-time decisions and active risk monitoring to enable their customers to … maximize profitability while managing risk. their solutions are continually adapting to changing market conditions. They 20+ years of retail data science expertise and experience. They work with a global base of leading commerce partners, representing a big proportion of all US omnichannel retail sales activity across hundreds of … to operations, finance, marketing, and loss prevention. Their performance-improvement solutions yield measurable results with significant return on investment. About the role The Data Engineer will be part of a team that follows DataOps best practices and owns the standardized data assets and the processes around more »

Data Engineer (Snowflake) - Up to £80K + Benefits – Flexible Working - London Overview: We are working with one of the market leaders and most exciting organisations within the retail sector. We’re seeking an experienced Data Engineer who is analytically focused with expertise and proven experience with … efficient databases to support internal enterprise applications. Collaborate closely with development and infrastructure teams to ensure seamless integration and optimal database performance. Lead data strategy aligned with organizational objectives, including data governance, architecture, quality management, and integration initiatives. Develop and support data warehouse, customer … data platform (CDP), and associated data pipelines using Matillion and Snowflake. Gather requirements from the Business Intelligence team and translate them into technical specifications for implementation by the software development team. Define software requirements for data maintenance and dashboard requirements for displaying insights to more »

Eames Consulting are delighted to be supporting an established Investment management company that are looking for a Data engineer. Responsibilities: Develop and maintain complex SQL queries, customize reports, and optimize data usage. Create tools to address business challenges and empower users with third-party tools. Design … and implement cloud-based data platforms, ETL pipelines, and data integration. Collaborate with colleagues to ensure data integrity, quality, and security standards are met. Experience required: 3 years experience working as a Data engineer or similar Strong proficiency in SQL or Data Bricks and database development, including complex query optimization and performance tuning. Comfortable using programming languages such as Python Strong Data Warehousing experience - building data pipelines from scratch Good understanding of open source, big data, and cloud data platforms (e.g. AWS more »

This innovative Multi-Strategy Hedge Fund, seek an experienced Software Engineer with expertise in Python to work within a dynamic Data Engineering Group building greenfield pipelines, frameworks, APIs and core Data store (Data warehouse and Data Lake) to enable the ingestion and … teams firmwide (X-Asset - Equities, Rates, Commodities, FX and Crypto). Key Skills required 3+ years of exp. as a Software Developer/Data Engineer with Python as your core language Extensive API and Microservices development experience 3+ years SQL Strong DB skills ideally in both SQL and more »

human feel of our match simulation. Areas you’ll be involved with include, Full Body Inverse Kinematics, Motion Matching, Skeleton Retargeting, Data Pipeline and Compression, Events Editor, Real-Life Match Data 3D Viewer, Statistics and Generative AI. Collaborate with other disciplines, including animations, artists, QA more »

Are you a Data Engineer passionate about the future of VR and AR? A leading global tech company is pushing the boundaries of what's possible, and they need you to join their cutting-edge Audio team! Our client is on a mission to create virtual sounds that … are indistinguishable from reality and redefine human hearing, enabling connections that transcend physical distance. Role : Data Engineer Location : UK – remote Salary Range : up to £65,000 doe and location Contract Length : 12 months – PAYE – within IR35 Candidates to be considered for this opportunity MUST hold an appropriate work … multimedia datasets using online workflows and command line tools/scripts. Manage and document multimedia datasets, collaborating with internal teams. Optimize data pipeline tooling, working closely with developers. What You Bring: Proven experience with large spreadsheets/databases (Excel, MySQL). Proficiency in Python (NumPy, pandas, SciPy more »

Data Software Engineer– London Sports Analytics & Statistics Company Fulltime onsite *Please note this role cannot sponsor. Please do not apply if you are seeking sponsorship* A Data Engineer is required for an exciting and innovative Software sports betting company based in London. The successful candidates will … researchers and alongside other extremely talented and driven engineers to build and support systematic trading models. You will be working on real-time data pipelines for live sports events where your expertise will help the company advance their in-play sports analytics and pricing function. Your day-to … day will involve developing robust and scalable data pipelines, implementing new feeds into the business, and integrating them with their quantitative decision models. Essential Skills Proven experience as a data engineer or similar, with a track record of handling mid to large scale data more »