26 to 50 of 75 Data Lake Jobs in the City of London

Social network you want to login/join with: Data Engineer - Pyspark/Palantir, City of London col-narrow-left Location: City of London, United Kingdom Job Category: Information Technology EU work permit required: Yes col-narrow-right Job Reference: BBBH63893_1748355123 Job Views: 5 Posted: 27.05.2025 Expiry Date: 11.07.2025 col-wide Job Description: Data Engineer … Pyspark/Palantir Whitehall Resources require a Data Engineer with experience with Pyspark & Palantir to work with a key client on an initial 6 month contract. *Inside IR35. *This role will require on site work in London 2-3 days per week. Data Engineer - Pyspark/Palantir Key responsibilities: • Developing Data Stores and Data Warehouse solutions • Design and develop data pipelines bring data from variety of data sources to Azure Databases and Azure Data Lake using self-integration runtime • Work with Product owners and Solution Architect to understand the requirements and deliver in Azure platform, • Design, Estimation and Planning • Executing the PoC/Project More ❯

result in low levels of production issues Awareness of industry best practice and new and upcoming technology trends Full solution architectural understanding across networking and software Experience of data programmes and data technologies, ideally gained within an enterprise scale organisation or within technology or development team or similar. Technical exposure to and/or experience in … a range of technologies and architectural best practice concepts, specifically Data … Engineering Design Snowflake expert with hands-on experience of data modelling and best practises MS Azure Data Stack - Data Factory, Data Lake Storage, Azure SQL, Azure ML, Power BI Automation of CICD pipelines across different technologies including Power BI, Snowflake and the Azure Data Stack Data Science More ❯

Provide architectural guidance on security, compliance, and governance in cloud solutions. Collaborate with stakeholders to create digital roadmaps for long-term cloud transformation. Design conceptual, logical, and physical data architectures that support business and analytics needs. Implement data cataloging and data governance frameworks to enhance data discovery, lineage tracking, and compliance. Define … data modeling standards, ensuring alignment with business intelligence, AI/ML, and analytics strategies. Develop data security frameworks, including encryption, access control, and data masking. Support data migration and transitioning strategies, ensuring minimal disruption to business operations. Optimize data storage and retrieval mechanisms across cloud-based databases, preferably Azure CosmosDB … GCP). Enterprise Architecture: Proven ability to design and implement end-to-end cloud solutions, including multi-cloud and hybrid models. Strong knowledge of conceptual, logical, and physical data modeling. Experience in data cataloging, data lineage, and governance frameworks. Hands-on experience with cloud-based databases, data lakes, and data More ❯

Our client a London based Technology and Data Engineering leader have an opportunity in a high growth AI Lab for an ‘AI Engineering Researcher' A UK based 'Enterprise' Artificial Intelligence organisation, focussing on helping accelerate their clients journey towards becoming 'AI-Optimal' - starting with significantly enhancing its abilities in leveraging AI & machine intelligence to outperform traditional competition. The … expanding research team of exceptional PhD computer scientists, software engineers, mathematicians & physicists, to use a unique multi-disciplinary approach to solving enterprise-AI problems. Principal Activities of role: Data Pipeline Development: • Design, develop, and maintain ETL processes to efficiently ingest data from various sources into data warehouses or data lakes. • Data Integration and Management: Integrate data from disparate sources, ensuring data quality, consistency, and security across systems. Implement data governance practices and manage metadata. • System Architecture: Design robust, scalable, and high-performance data architectures using cloud-based platforms (e.g., AWS, Google Cloud, Azure). • Performance Optimization: Monitor, troubleshoot, and optimize data More ❯

login/join with: Duration: contract to run until 31/12/2025 Rate: up to £644 p/d Umbrellainside IR35 We are seeking an experienced Data Modeller with proven expertise in the London Market insurance sector. The successful candidate will play a key role in designing and validating data models that support enterprise … data initiatives. This includes working closely with data engineers, architects, and business stakeholders to ensure data structures are scalable, accurate, and aligned with business needs. Key Skills/requirements Design and maintain conceptual, logical, and physical data models to support reporting, analytics, and operational systems. Collaborate with data engineers and … to explain complex data concepts to non-technical stakeholders. Preferred Skills: Experience with insurance platforms such as Guidewire, Duck Creek, or legacy PAS systems. Knowledge of Delta Lake, Apache Spark, and data pipeline orchestration tools. Exposure to Agile delivery methodologies and tools like JIRA, Confluence, or Azure DevOps. Understanding of regulatory data requirements More ❯

A leading, technology-focused, multi-strategy Hedge Fund are seeking a Senior Data Engineer (Front Office) to join their innovative and highly experienced team Essential Skills: 5+ years of experience with a proven track records of building data platforms Expert-level Python and SQL skills (C++ experience is a plus, but not essential) Interest in contributing … merging technology and research to solve some of the most complex challenges and deliver high-quality returns. This flexible role (2 days WFH) involves building and supporting bespoke data solutions for Quants and Traders, and developing a scalable, global Data Lakehouse used by multiple trading … desks. You’ll design and implement data pipelines in Python within an AWS cloud environment, working on data modelling for large-scale Data Lake and OLAP systems. Key responsibilities will include optimising performance in cross-region setups, ensuring data completeness and quality, and contributing to the software and infrastructure design. This More ❯

Azure Data Architects x 2 The Company: At Amber Labs, we are a cutting-edge UK and European technology consultancy that prioritises empowering autonomy, promoting experimentation, and facilitating rapid learning to provide exceptional value to our clients. Our company culture is centred around collaboration, where all colleagues, regardless of their role, work together to minimise risk and shorten … and support staff. Mostly remote some Ad hoc travel (London) Eligble for BPSS or SC clearance Digital Transformation Project About the Role: We are seeking highly skilled Azure Data Architects who have worked for the NHS to join our dynamic team. In this role, you will be responsible for designing and implementing end-to-end cloud solutions on … such as: Azure Synapse Analytics Azure Data Factory Azure Databricks Azure SQL & Cosmos DB Azure Purview (for Data Governance) Azure Blob Storage & Data Lake Azure Event Hubs & Streaming Analytics Hands-on experience with data integration, ETL/ELT pipelines, and batch/real-time processing. Strong knowledge of data More ❯

Data Platforms Product Owner | Technical Data Background | Banking/Cards/Insurance | London (hybrid) | £600/day (Inside IR35) | 6 month initial contract Our London based Financial Services client is looking for a Technical Data Platforms Product Owner to work on a large project. You'll have a solid data/tech … background, preferably in Financial Services. You'll have the ability to write the User Stories for the Data Engineers and tell the Architect what you need in the Data Model - and you'll define the data platforms roadmap. You will also have good experience around ADF, Azure DataBricks, and Snowflake - plus your knowledge will … include ETL processes, Data Governance, Data Warehouses, Data Marts, and Data Lakes. Key Skills & Experience: Product Owner Data Platforms Roadmap User Stories Snowflake Azure Data Factory/Azure Data Bricks Data Models Great Communication skills This role would be based in the City of More ❯

Are you a Data Engineer who is looking for a more purposeful career? Feel like your current role, limits you to what you can achieve and that there isn’t any more room to grow? If the answer is yes, I could have just the role for you. Recruit with Purpose are supporting one the UK’s largest … Design and maintain scalable, high-performance data pipelines using Azure Data Platform tools such as Databricks (PySpark), Data Factory, and Data Lake Gen2. Develop curated data layers (bronze, silver, gold) optimised for analytics, reporting, and AI/ML, ensuring they meet performance, governance, and reuse standards. Translate business needs … architecture diagrams, lineage, and testing protocols. What they are looking for from you: High level of experience with Microsoft Azure (Azure Data Factory, Databricks, Data Lake Gen 2) Experience building stable and accurate data pipelines Expertise with SQL and Python Familiarity with medallion architecture 4+ years working in a fast-paced data More ❯

Data Scientist Hybrid – London (3 days a week) Up to £75,000 + Pension + Share Scheme About the Role We’re working with a professional services firm undergoing a major data transformation, now looking to bring on board their first in-house Data Scientist . Reporting directly into the CFO, you’ll take … the lead on developing and managing the firm’s data infrastructure, starting with the continued build-out and integration of their data lake. You’ll work on high-impact reporting and analytics projects, and long-term, deliver commercially focused data science solutions such as price optimisation and revenue forecasting. This is a hands-on … uncover opportunities to use data more effectively across the organisation. Key Responsibilities Build and manage data pipelines and infrastructure, including a developing data lake Support reporting projects and ongoing Power BI development Deliver commercial data science solutions (e.g., forecasting, optimisation) Integrate data from a variety of internal systems and More ❯

Technical Architect - Data Engineering/Azure Contract 6 months Rate £400 per day outside IR35 Location: London Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. Fractal is building a world where individual choices, freedom, and diversity are the greatest assets. An ecosystem where human imagination … technologies. Exposure across all the SDLC process, including testing and deployment. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Good knowledge of Python and Spark are required. Experience in ETL & ELT Good understanding of one scripting language Good … Ops, ML Ops, Deep expertise in Azure Databricks , ETL frameworks. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any More ❯

Senior Data & AI Consultant Homebased - Must be happy to travel to customer sites and London Full-time, Permanent Eligible for SC Clearance/Actively SC Cleared Be part of a growing consulting team with a leading Microsoft Partner. This role is for a Senior Azure Data & AI Consultant, who will deliver high-impact solutions that transform … You’ll play a critical role in guiding organisations through the AI journey — from identifying opportunities to architecting and deploying real-world solutions using modern cloud platforms and data technologies. You’ll work in a dynamic, fast-paced environment alongside experienced consultants and technical teams, helping clients unlock the full value of AI through innovation, collaboration, and expertise. … Looking For Proven experience delivering AI and data solutions in a Microsoft Partner or specialist technology consultancy environment Strong technical proficiency across Azure Synapse , Data Lake , Databricks , Power BI , Microsoft Fabric , and relevant AI services (e.g. Azure OpenAI, ML Studio) Deep understanding of modern data architectures, data governance, and operationalisation of More ❯

Join the Merlin Entertainments Digital & Data Team – where technology, data, and digital innovation meet magic! At Merlin Entertainments, we're transforming the entertainment and hospitality industries through superb digital experiences. Our Digital & Data function is connecting our global technology estate, delivering digital experiences to enchant more than 60 million guests in over 20 countries … and in-park interactions, improving global operations, or directly supporting guests by keeping attractions running smoothly. If you're a hands-on, passionate professional ready to modernise technology, data, and digital products while crafting memorable guest experiences, we'd love to hear from you! The Orchestration and Data Movement Service Systems Analyst is responsible for ensuring … the efficient and seamless movement of data across systems, applications, and environments, with a focus on orchestration. This role plays a key part in designing, implementing, and maintaining automated workflows that facilitate data integration, processing, and delivery. The Analyst will ensure that data movement processes are efficient, secure, and compliant with organisational standards while More ❯

Senior Data Engineer London/Hybrid 2 days per week Permanent Salary: Up to £90,000 We have partnered with a leading investment firm who are seeking a Senior Data Engineer. They have garnered a strong reputation in delivering solutions across the full investment life cycle. They are looking for a skilled and hands-on Senior … Data Engineer to design and build scalable data pipelines to support evolving business and technology needs. This is a key role and will contribute significantly to the company's vision of becoming a data-centric organisation. The ideal candidate will have demonstrable experience with the modern Azure tech stack and a deep understanding of … ingesting and transforming data from diverse internal and external sources. Key Responsibilities: Design and implement robust data solutions to support a modern cloud-native data warehouse. Provide technical leadership and identify effective solutions for complex data-related challenges. Perform and participate in peer reviews of design, code, and tests, ensuring high-quality More ❯

business problems using creative solutions, then we wish to talk with you. As a Lead Architect (Azure), you will work in the Technology team that helps deliver our Data Engineering offerings at large scale to our Fortune clients worldwide. The role is responsible for innovating, building and maintaining technology services. Responsibilities: Be an integral part of large-scale … technologies. Exposure across all the SDLC process, including testing and deployment. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Good knowledge of Python and Spark are required. Experience in ETL & ELT Good understanding of one scripting language Good … Ops, ML Ops, Deep expertise in Azure Databricks, ETL frameworks. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any More ❯

business problems using creative solutions, then we wish to talk with you. As a Lead Architect (Azure), you will work in the Technology team that helps deliver our Data Engineering offerings at large scale to our Fortune clients worldwide. The role is responsible for innovating, building and maintaining technology services. Responsibilities: Be an integral part of large-scale … technologies. Exposure across all the SDLC process, including testing and deployment. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Good knowledge of Python and Spark are required. Experience in ETL & ELT Good understanding of one scripting language Good … Ops, ML Ops, Deep expertise in Azure Databricks , ETL frameworks. Expertise in Microsoft Azure is mandatory including components like Azure Data Factory, Azure Data Lake Storage, Azure SQL, Azure DataBricks, HD Insights, ML Service etc. Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any More ❯

Principal Data Engineer £95,000 - £105,000 + Bonus London (other UK locations available) A top-tier global consultancy is in search of a Data Engineer (Principal Consultant) to drive the delivery of cloud-native data platforms across complex, high-impact environments. If you’re passionate about data engineering, experienced in building …/RFP), and want to work in an organisation that values creativity, autonomy, and technical excellence — this could be the one. What you’ll do as a Principal Data Engineer: • Lead the architecture and build of modern data platforms using AWS-native tools (S3, Redshift, Lambda, Glue, DynamoDB, Matillion, etc.). • Drive best practices in CI …/CD, Infrastructure-as-Code, and data ops methodologies. • Liaise directly with clients to define requirements, shape technical solutions, and oversee successful delivery. • Contribute to practice growth through recruitment, coaching, and mentoring. • Lead business development activities including RFI/RFP responses and proposal development. • Ensure solutions are secure, scalable, and production-ready. • Be an advocate of clean code More ❯

Job Title: Solutions Architect (Data) Terms: Permanent, full-time Salary: c.£75,000 Employee Benefits: Local Government pension scheme, 31 days annual leave (plus Bank Holidays) Allen Lane is delighted to be supporting a London-based Local Authority in their search for a Solutions Architect . This is a new role in the structure, with a particular focus … on data architecture and data platforms . The postholder will own end-to-end solution designs, working closely with senior stakeholders cross the Council. We are looking for applicants that can demonstrate outstanding collaboration and engagement skills, communicating effectively will a diverse range of stakeholders up to and including Director level and with external partners. You … will play a critical role in leading the definition, design and delivery of modern data solutions and products. You will have excellent Cloud knowledge, either with Azure or AWS . Applicants are expected to demonstrate the following skills and experience against the following essential criteria: Relevant certifications (e.g. Azure Solutions Architect, Azure Data Scientist, Azure AI More ❯

Data Engineer (one day a week in the central London office) Sports Analytics This role is a unique opportunity for a Data Engineer to combine technical challenges with creativity in a collaborative, high-standard work environment. By joining this team, you’ll not only be part of a creative and open work culture focused on innovation … it an ideal setting for those passionate about leveraging cutting-edge technology to make meaningful contributions in the world of sports analytics. Key responsibilities for the role of Data Engineer include: Design, construct, install, test, and maintain highly scalable data management systems. Ensure systems meet requirements and industry practices for data quality and integrity. … workflows with Composer or Vertex Pipelines. Data Architecture & Modelling: Ability to translate diverse business requirements into scalable data models and architect a data lake/lakehouse environment on GCP. Engineering Best Practices: Proficiency with Git, CI/CD for data systems, and robust testing methodologies. Highly desirable skills include: Domain Experience More ❯

Data Engineer (one day a week in the central London office) Sports Analytics This role is a unique opportunity for a Data Engineer to combine technical challenges with creativity in a collaborative, high-standard work environment. By joining this team, you’ll not only be part of a creative and open work culture focused on innovation … it an ideal setting for those passionate about leveraging cutting-edge technology to make meaningful contributions in the world of sports analytics. Key responsibilities for the role of Data Engineer include: Design, construct, install, test, and maintain highly scalable data management systems. Ensure systems meet requirements and industry practices for data quality and integrity. … workflows with Composer or Vertex Pipelines. Data Architecture & Modelling: Ability to translate diverse business requirements into scalable data models and architect a data lake/lakehouse environment on GCP. Engineering Best Practices: Proficiency with Git, CI/CD for data systems, and robust testing methodologies. Highly desirable skills include: Domain Experience More ❯

Hybrid | 3 days/week working from office) Duration: 6 months Contract Skill Sets: • Palantir Foundry • ETL • Spark • Python • PySpark • Informatica • AWS • SQL, PLSQL • Shell scripting • Data Lake • Data warehousing • Scala • Oracle • MS SQL Server • PowerBI. Thanks & Regards Milan | Talent Acquisition | Europe & UK Ampstek Services Limited Kemp House, 152-160, City Road London-EC1V 2NX More ❯

Engineer for Backend services you will be responsible for a range of backend systems such as Software applications that include areas such as: Payment Gateway, Settlement, Transaction Fraud Monitoring, Datalake Service and the Core Banking System. You will oversee their deployment, development, enhancements and production operations. You should have extensive experience and skills in Software Engineering, DevOps, project management, incident More ❯

Are you a passionate Data Engineering Manager ready to unlock the power of data? Our client is a leading UK-based consultancy seeking a skilled professional to shape data strategies, mentor dynamic teams, and deliver cutting-edge solutions. With hands-on expertise in Spark, SQL, and cloud platforms like Azure, you’ll lead end … to-end projects, drive innovation, and collaborate with clients across industries. What You’ll Do: Lead complex data engineering solutions with technical excellence. Mentor teams and foster a culture of growth and quality. Showcase your expertise in pre-sales and client workshops. Design data pipelines, lakes, and warehouses with best practices. Skills They're Looking For … Proven Data Engineering experience (5+ years). Consultancy experience is a must. Leadership and multi-project environments experience. Expertise in ETL, data modelling, and Azure Data Services. Experience in designing and implementing data pipelines, data lakes, and data warehouses. Hands-on experience with Apache Spark and bonus points More ❯

Social network you want to login/join with: Data Engineering Manager/London/Microsoft Gold Partner, london (city of london) col-narrow-left Client: Opus Recruitment Solutions Location: london (city of london), United Kingdom Job Category: Other - EU work permit required: Yes col-narrow-right Job Views: 2 Posted: 16.06.2025 Expiry Date: 31.07.2025 col-wide Job … Description: Are you a passionate Data Engineering Manager ready to unlock the power of data? Our client is a leading UK-based consultancy seeking a skilled professional to shape data strategies, mentor dynamic teams, and deliver cutting-edge solutions. With hands-on expertise in Spark, SQL, and cloud platforms like Azure, you’ll lead … end-to-end projects, drive innovation, and collaborate with clients across industries. What You’ll Do: Lead complex data engineering solutions with technical excellence. Mentor teams and foster a culture of growth and quality. Showcase your expertise in pre-sales and client workshops. Design data pipelines, lakes, and warehouses with best practices. Skills They're Looking More ❯

Data Engineer Location: London (4 days/week on-site) Type: Contract (Outside IR35) Duration: 6 months A leading physical commodities trading firm is seeking a Data Engineer for a 6-month contract (Outside IR35). You’ll work on-site in London 4 days per week, building and optimizing data pipelines in a … fast-paced, data-driven trading environment. Responsibilities: Build and maintain scalable data pipelines (ETL/ELT) Support data users with ingestion, debugging, and navigation Integrate data from diverse sources (structured & unstructured) Monitor and ensure data quality Maintain and improve existing production processes Develop dashboards and visualizations Work closely with data scientists and stakeholders Follow CI/CD and code best practices (Git, testing, reviews) Tech Stack & Experience: Strong Python (Pandas), PySpark, and SQL skills Cloud data tools (Azure Data Factory, Synapse, Databricks, etc.) Data integration experience across formats and platforms Strong communication and data literacy Nice to Have: Commodities/ More ❯