16 of 16 Data Lake Jobs in the City of London

data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous ...

This is a rare opportunity to apply serious data engineering in a domain where latency, correctness, and reliability carry direct commercial weight. * Requirements * 6+ years data engineering in production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts * ETL/ELT pipeline design … retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade ...

distribution keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue …/Iceberg for low-cost, serverless analytics. Manage Glue Data Catalog integrations and table schema management. 5. Storage (S3) & Data Lake Architecture Design secure, cost-efficient S3 data lake structures (bronze/silver/gold zones). Implement data lifecycle ...

Responsibilities Design, build, and maintain scalable data pipelines (primarily in Python) Architect and evolve a modern data platform (ingestion, lake, warehouse, and consumption layers) Develop and optimise data warehousing solutions, including ClickHouse and cloud-based platforms Implement and manage data orchestration … practices Experience with semantic layers and business-facing data models Track record of building end-to-end data platforms (ingestion → lake → warehouse) Comfortable working in an agile, iterative delivery environment The Offer Salary: Up to £110,000 Bonus: Strong performance-based bonus Location: London (Hybrid ...

critical functions. Responsibilities: Design, build and deploy scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality … business teams to align engineering solutions with organisational needs. Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. ...

full of opportunities to turn today’s challenges into tomorrow’s solutions. Job Description Becoming part of the UK team in our Data & AI Service Line will present new opportunities for you to make a positive impact as we embark on securing major projects with our top international … working with our FTSE100 clients and beyond on a wide range of advisory projects, bringing deep business and technology expertise to help address Data and AI opportunities and challenges across their organisations. Wavestone has a significant and growing capability in Data & AI that is in high ...

Data Bricks Product Owner, Microsoft Fabric Product Owner, Data Engineering, AI Data Analytics, Microsoft Stack, Azure, Data Bricks, ML, Azure, Mainly Remote Data Bricks Product Owner, Microsoft Fabric Project Owner, Data Engineering/Technology Product Owner required … Engineering (Data Bricks, Microsoft Fabric etc) before moving into Product Ownership Technical proficiency:Familiarity with Azure services (e.g., Data Lake, Synapse, Fabric) and Databricks for data engineering, analytics, performance optimisation, and governance Development Framework experience within Microsoft Stack Technologies Experience with implementing ...

Data Engineer Contract Duration: 4 months Location: Farringdon, London Job Type: Contract, Hybrid (3 days a week onsite) We are seeking a Data Engineer to our client's ambitious technology transformation. This role is crucial in implementing the end-to-end delivery of data strategies, working closely with Data Analytics and Data Science teams to ensure best practices and standards are met. Day-to-day of the role: Source, map, and extract data from various business and cloud-based source systems. Design, develop, test, optimise, scale ...

Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time on-site when required Expertise required in AWS, Data Pipelines, ETL, Data … analytics and reporting. Develop ETL/ELT workflows to move data efficiently into data warehouses, data lakes and lake houses using open-source and AWS tooling. Apply DevOps practices, including CI/CD, infrastructure as code and automation, to improve and streamline data ...

other Azure services). Strong Power BI capability (reporting + modelling) and ability to connect engineering outputs to analytics consumption. Understanding of Direct Lake semantic modelling design considerations (model creation, table choices, and enforcing access rules). Proven experience with ingestion and transformation patterns (ETL/ELT concepts), data … Experience across adjacent Microsoft services often paired with Fabric (e.g., Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure Data Lake). Familiarity with Microsoft Purview and broader governance practices. Dynamics 365 CE and FO integration and analytics. Exposure to AI/ML enablement ...

successful multinational technology business is looking for a Data Reliability Engineer to join its growing data team in Central London. This role is hybrid youll be able to work from home 2 days per week. This is a high-impact role focused on improving data quality, reducing incidents, and building scalable observability across a modern enterprise data platform. Youll help ensure data across the organisation is accurate, reliable, and trusted for critical business decision-making. Youll take ownership of data reliability end-to-end, designing and implementing ...

Senior Data Architect Location: Remote (UK-based) Employment Type: Permanent Sector: Consulting/Public Sector Eligibility: Applicants must have the existing right to work in the UK. Visa sponsorship is not available. Overview We are seeking a Senior Data Architect (20+ years’ experience) to lead … design and delivery of enterprise-scale data platforms within consulting and public sector environments. This role requires strong architectural leadership, stakeholder engagement, and deep expertise in modern data platforms that support advanced analytics and AI-enabled solutions. Key Responsibilities • Lead the architecture and design of enterprise ...

Senior ML/MLOps Engineer (MLflow) | Databricks | Azure) About Optima Partners We’re an advanced data & business consultancy helping major consumer brands transform through data-driven customer strategy, design, and engineering. Our clients include leading UK and global brands across financial services, retail, and more. Join … workflows Build scalable data & ML pipelines in Databricks (Delta, DLT, Jobs) Design and maintain Azure data architectures (Data Lake, Synapse, SQL DW) Create robust CI/CD pipelines for ML + data using Azure DevOps, Git, Jenkins Automate infrastructure with Terraform ...

step directly into a fast paced environment that moves like a FinTech despite being in the Trading industry. Someone who has experience uploading data to databases using Python and writing using object-oriented programming as well … massive necessity. There is no Financial Services industry knowledge needed but it is beneficial. They are planning to build their own data lake internally and more importantly need someone with strong understanding of how to do this, as well as, build quickly and be able to utilise ...

hiring a Solutions Architect to design scalable, secure and data driven solutions across enterprise and government programmes. This role is ideal for someone who enjoys shaping architecture from early presales through to delivery, working closely with customers and engineering teams to build real world impact. Company Description Goaco … consultancy delivering secure, user-centered, and innovative digital solutions to public and private sector organisations. The company specialises in digital transformation, cyber security, data & AI, software development, and cloud services. Since its establishment in 2010, Goaco has built a trusted reputation by working with over 100 subject-matter ...

design, configuration, and implementation within an international investment-banking environment. Systems Expertise: Deep practical knowledge of Oracle ERP/EPM modules, financial data integration, reporting tools, API-based integrations, ETL technologies, data lakes, accounting rules engines, and reference data. This is a fascinating role with ...