1 to 25 of 30 Permanent Data Lake Jobs in the East of England

Data Engineer £53,000 per annum Bedford Permanent Hybrid Working Role Overview Join us at the Money and Pensions Service (MaPS) as a Data Engineer where you will design and implement data pipelines to connect operational systems, engineering data for analytics … profiles and establish recovery processes to maintain data integrity. Azure Services • Utilise Cloud based data technology platforms, Data Lake based storage & Cloud data pipeline/orchestration tools for efficient data storage, processing, and transfer.• Ensure the scalability and reliability ...

regulatory standards. Proficiency in AWS ETL technologies, including Glue, Data Sync, DMS, Step Functions, Redshift, DynamoDB, Athena, Lambda, RDS, EC2 and S3 Datalake, CloudWatch for building and optimizing ETL pipelines and data migration workflows. Working knowledge of Azure data engineering tools, including ADF (Azure … Data Factory), Azure DB, Azure Synapse, Azure Data lake and Azure Monitor providing added flexibility for diverse migration and integration projects. Prior experience with tools such as MuleSoft, Boomi, Informatica, Talend, SSIS, or custom scripting languages (Python, PySpark, SQL) for data extraction ...

distribution keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue …/Iceberg for low-cost, serverless analytics. Manage Glue Data Catalog integrations and table schema management. 5. Storage (S3) & Data Lake Architecture Design secure, cost-efficient S3 data lake structures (bronze/silver/gold zones). Implement data lifecycle ...

Data Engineer This role is largely remote with the occasional across sites when required … seeking an experienced Data Engineer who is skilled in all toolsets in data platforms including Azure SQL, data lake, databricks and data factory, and specialised in building high performance, highly scalable and cost-effective data solution within time ...

Data Architect - £60,000-£80,000 | Hybrid (Hatfield, 2 Days Onsite) | January Start A leading UK utility company is seeking an experienced Data Architect to join their team and play a pivotal role in shaping and governing their enterprise data landscape. This … . Embed data governance and data quality frameworks across pipelines. Work with the existing technology stack (Redshift Data Lake, Talend, Oracle, QlikView) while leveraging knowledge of modern platforms such as Snowflake, Databricks, Synapse. Engage with senior stakeholders and lead architectural decisions that align ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

leading data consultancy is seeking an experienced Senior Data Engineer to join their team and deliver cutting-edge solutions for a major transportation project. This is an exciting opportunity to work on a high-impact initiative for a well-known UK transport provider, leveraging modern cloud … Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice-to-Have Experience with Scala for Spark development. Knowledge of CI/CD pipelines ...

reporting engineer to enhance their career with a modern business who have big plans for 2026. Power BI, Power Platform, Azure Data Lake and Data Services experience are all essential qualities to apply for this role. In this role you will be responsible for designing … driven policy. Key Skills Requirements Include: Strong experience with Power BI Key understanding of data sourcing inc. Data Lake, Power Platform and SQL Familiarity with Data Factory Understanding of CI/CD for Power BI Keen interest in the future ...

research and education as we look to establish and develop a hub for our wide-ranging innovation work across the Trust. The Research Data Engineer is a key role in this new Unit, contributing to the delivery of advanced data engineering solutions to manage large volumes … data effectively. The role has a particular focus on supporting the technical delivery and implementation of key commercial partnerships, curating viable, deep, longitudinal and anonymised research datasets, with an initial focus on oncology. In doing so, you will support the generation of new insights and cutting-edge applications ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

continually refine the AI initiatives portfolio and roadmap for the group. Run a structured intake process for new ideas, assessing value, feasibility, data readiness, ethical risk, and compliance. Define clear success measures for every proof of concept (PoC) and MVP, including KPIs, leading indicators, and decision gates … such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow/ ...

Role: BI and Data Engineer Location: Norwich Salary: £60,000 Flexible working: 1-2 days a week in the office My client, a specialist technology organisation, is looking for a BI and Data Engineer to join their growing team. This is an exciting opportunity to work … skills/Experience Own automation/testing for Power Platform roll-out. Power Platform (Power BI, Power Apps, Power Automate) Azure Data Lake, Synapse, Data Factory Comfortable with working data pipelines Experienced working with SSRS Financial services experience would be advantageous If this ...

seeking a highly skilled Business Intelligence & Power BI Developer to design, develop, and maintain high-quality reporting and dashboard solutions that support data-driven decision making across the organisation. Working closely with senior finance stakeholders, you will transform complex data into meaningful insights and ensure robust … skills and experience with data cleansing and modelling Experience working with Azure/Microsoft BI technologies (SQL Server, Data Lake, SSIS/SSAS/SSRS) Familiarity with DevOps and Agile delivery methods (Azure DevOps, CI/CD, Scrum/Kanban) Experience with other BI tools ...

Data Architect 24-Month Fixed Term Contract£60,000-£80,000Hatfield (Hybrid, 2 days per week onsite)As the first Data Architect in the function … will set the standard for data architecture across the organisation. You Technical Environment Current in-house technologies include:* Redshift Data Lake* Talend* Oracle* QlikView You will also need working knowledge of modern data technologies such as Snowflake, Databricks, Synapse or similar. What ...

Architect to lead the design, development, and optimization of enterprise-scale BI solutions. This role requires deep technical expertise in Power BI, data modeling, and performance tuning, especially for large and complex datasets. The ideal candidate will be proficient in tools like DAX Studio, Tabular Editor, and Azure … datasets in Power BI and Azure. Solid understanding of SQL, ETL processes, and data warehousing. Familiarity with Azure Synapse, Data Lake, and Analysis Services (AAS/SSAS). Strong performance tuning: aggregations, incremental refresh, Hybrid tables, Direct Query optimizations, query folding Benefits : Competitive salary ...

Architect to lead the design, development, and optimization of enterprise-scale BI solutions. This role requires deep technical expertise in Power BI, data modeling, and performance tuning, especially for large and complex datasets. The ideal candidate will be proficient in tools like DAX Studio, Tabular Editor, and Azure … datasets in Power BI and Azure. Solid understanding of SQL, ETL processes, and data warehousing. Familiarity with Azure Synapse, Data Lake, and Analysis Services (AAS/SSAS). Strong performance tuning: aggregations, incremental refresh, Hybrid tables, Direct Query optimizations, query folding Benefits: Competitive salary ...

Senior Data Engineer | Remote (UK or Europe) A leading prop trading firm is hiring a Senior Data Engineer to take ownership of their trading and market data infrastructure. Operating globally with connectivity to dozens of exchanges, the firm executes millions of trades daily with … data playing a mission-critical role in every aspect of the business. This is a high-impact, high-autonomy role in a lean engineering environment. You'll design and maintain real-time and historical data systems, power internal analytics tooling, and ensure data quality ...