101 to 125 of 136 PySpark Jobs in the UK

Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance, and Solvency II teams to meet Lloyd's market … highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing CI/CD pipelines (Azure DevOps, branching/merging, automation). Solid knowledge of data governance, lineage, access ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

lakes, lakehouse, distributed systems) Exposure to proof of concept (PoC) development Knowledge of object storage and distributed processing frameworks Familiarity with technologies such as: PySpark, Dremio, Apache Iceberg, Apache Airflow, S3/object storage concepts, IBM Cloud (or willingness to upskill) Experience in financial services or highly regulated environments ...

orchestrate data pipelines across ingestion, transformation, and modelling layers Optimise and monitor data workflows using Data Lakes and Logic Apps Advanced SQL, Python & PySpark Write and maintain complex T-SQL queries, stored procedures, and optimised database logic Use Python and PySpark for distributed data processing and large-scale … Microsoft Fabric (Lakehouse, pipelines, notebooks) Strong understanding of Data Lakes and Logic Apps Advanced metadata-driven ETL/ELT pipeline development Strong Python and PySpark capabilities Advanced T-SQL and SQL performance optimisation Experience with Azure DevOps (Boards, Repos, Deployment Pipelines) Desirable skills Experience with SSIS and SSAS Tabular ...

data pipelines Contribute to the development of a Microsoft Fabric data platform Work with Lakehouse/medallion architecture Develop ETL/ELT pipelines using PySpark and SQL Ingest and transform data from SQL systems, flat files, and CSV sources Collaboration & Stakeholder Engagement Partner with teams across the organisation … reporting teams with reliable, well-structured datasets Contribute to data modelling and architectural decisions Technology Stack The current data platform includes: Microsoft Fabric PySpark T-SQL Lakehouse/Medallion architecture Power BI Current workloads are primarily batch-based, with ambitions to move towards streaming capabilities in the future. Essential ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

PySpark Engineer Lead As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift you will refactor complex procedural logic into scalable, production-ready distributed pipelines … Tier-1 financial services environment. Core Responsibilities Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark ...

PySpark Engineer Lead As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift you will refactor complex procedural logic into scalable, production-ready distributed pipelines … Tier-1 financial services environment. Core Responsibilities Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark ...

Lead PySpark Engineer (Cloud Migration) Role Type: 5-Month Contract Location: Remote (UK-Based) Experience Level: Lead/Senior (5+ years PySpark) Role Overview We are seeking a Lead PySpark Engineer to drive a large-scale data modernisation project, transitioning legacy data workflows into a high-performance … cloud environment. This is a hands-on technical role focused on converting legacy SAS code into production-ready PySpark pipelines within a complex financial services landscape. Key Responsibilities Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog . Collaborate closely with insurance domain teams- Underwriting, Actuarial, Delegated Authority, Bordereaux, Exposure Management, Reinsurance, Finance, Risk and Solvency II -delivering solutions aligned … Azure data services and Databricks in enterprise-scale environments. Strong knowledge of Delta Lake , Medallion architecture, distributed compute and Lakehouse engineering patterns. Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps CICD , branching/merging and automated testing. Solid understanding of data governance, lineage ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

experience to Design & Develop a ETL pipelines in aws Glue for ingesting data from sql server or other sources Hands on experience on PySpark & Python Experience of implementing change data capture pipelines using aws dms for near-real time or batch ingestion Experience of developing incremental loads Glue … experience to Design & Develop a ETL pipelines in aws Glue for ingesting data from sql server or other sources Hands on experience on PySpark & Python Experience of implementing change data capture pipelines using aws dms for near-real time or batch ingestion Experience of developing incremental loads Glue ...

Senior Data Engineer (Python/PySpark & SQL) London (Hybrid - 3 days in office | Monday, Tuesday, Wednesday) Up to £85,000 dependent on experience Ref: J13073 We're working with a fast-growing AI SaaS business where data sits at the core of everything they build. They're looking … working with real-world, messy data and shaping it into reliable, decision-ready datasets used across product, analytics, and machine learning. Python/PySpark and SQL are central to the role. The Role: ·You'll design and maintain data pipelines, handle edge cases, and ensure data quality ...

value from commercial data across Foundry. Key Responsibilities: Architect end-to-end AI solutions using Foundry, including machine learning models, large-scale data pipelines (PySpark), and production-grade deployment architectures within Foundry’s ecosystem. Evaluate and select optimal ML and data technologies, frameworks, and platform components within Foundry … degree in Computer Science, Engineering, or related field. Proven experience as an AI Architect, ML Engineer, or similar role implementing solutions on Foundry, PySpark, and cloud-based big data environments. Deep understanding of AI/ML algorithms, model optimization, and enterprise-grade model deployment (preferably via Foundry). Experience ...

requiring strong hands-on expertise, proactive risk management, and excellent stakeholder communication skills. Key Responsibilities Architect and implement data solutions using Databricks on AWS, PySpark, and Python Lead cloud migration initiatives, including on-prem to AWS migrations Drive GenAI-enabled automation for data asset migration Develop scalable data models … deliver actionable business insights Design and optimize ETL pipelines (Informatica or similar tools) Build and deploy PySpark applications for large-scale data processing Work on LLM pretraining, fine-tuning, augmentation, and optimization Ensure high data quality through cleaning, transformation, and validation (Python/SQL) Automate data workflows to enhance ...

translating ERP data into scalable BI models Building robust Power BI datasets, reports, and semantic models Delivering complex data transformations using Python and PySpark Writing advanced T-SQL queries and performance-optimised stored procedures Acting as a senior escalation point for BI developers and coordinating workload distribution Ensuring governance … skills across modelling, DAX, optimisation, and dataset design Strong SQL capabilities (T-SQL, performance tuning, stored procedures) Experience with Python and/or PySpark for data transformation Background coordinating or leading BI developers Experience delivering BI in enterprise, multi-region environments Excellent communication skills, analytical mindset, and ability ...

orchestration, experiment tracking, hyper-parameter tuning, and deploying automated model retraining pipelines. Programming & Data Engineering : Proficiency in object-oriented Python, advanced dataframes (Polars/Pyspark), and data versioning (DVC). Experience designing data storage solutions and using object-oriented SQL interfaces. Cloud & DevOps : Hands-on experience with at least … experience Advanced Education : A Master's degree or PhD in a relevant field is a strong plus. Parallelisation & Performance : Experience with parallelisation frameworks like Pyspark or Ray. Advanced Cloud & Infrastructure : Familiarity with serverless deployments (e.g., Fargate, Lambdas), infrastructure automation with Terratest or Ansible. ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience Hands-on experience with Microsoft Fabric, Databricks … equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Support data integration, transformation, and performance optimisation Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience … Hands-on experience with Microsoft Fabric, Databricks, or equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling Experience designing efficient, scalable data architectures ...