101 to 125 of 143 PySpark Jobs in the UK

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

modernisation of existing data estates. 🔧 Key Responsibilities Design and build scalable data pipelines using Databricks Develop and optimise ETL/ELT workflows using Spark (PySpark/SQL) Support migration and modernisation of legacy data platforms into Databricks Improve performance, reliability and cost efficiency across data workloads Work closely with … best practices across delivery ✅ What We’re Looking For Strong hands-on experience with Databricks in a Data Engineering capacity Solid experience with Spark (PySpark/Spark SQL) Proven track record building and optimising data pipelines (ETL/ELT) Experience working within cloud environments (AWS or Azure) Strong ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third‐party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd’s Syndicates Delegated ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

Description Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

We’re looking for a highly skilled Lead Data Engineer to provide technical leadership across data engineering, delivery execution and project governance. This role ensures development standards are upheld while delivering high‐quality, reliable solutions ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

reduce false positives/negatives, and support downstream fraud and KYC models. Build and maintain scalable data pipelines and feature stores in Spark/PySpark (or Scala), including data normalization, deduplication, and feature computation across large PII datasets in AWS/Databricks environments. Lead A/B tests … proficiency in Python (preferred) or Scala, including experience with ML libraries such as scikit‐learn, XGBoost, TensorFlow or PyTorch. Extensive experience with Spark or PySpark and distributed data systems (e.g., AWS EMR, Databricks) working on very large, messy datasets. Deep understanding of supervised and unsupervised learning, feature engineering, model ...

Role GenAI Data Engineer Your responsibilities: Design and maintain scalable data pipelines using PySpark, Python, and distributed computing frameworks to support high‐volume data processing. Architect and optimize AWS-based data and AI infrastructure, ensuring secure, performant, and cost‐efficient ingestion, transformation, and storage. Develop, finetune, benchmark, and evaluate … capabilities into production systems and ensure high-quality data, governance, and operational reliability Your Profile Essential skills/knowledge/experience: Strong experience with PySpark, distributed data processing, and largescale ETL/ELT pipelines. Strong SQL expertise including star/snowflake schema design, indexing strategies, writing optimized queries ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

Skills and Experience Essential Skills Strong hands‐on experience with Palantir Foundry , including Code Repositories, Workshop, AIP Logic, and Automate Proficiency in Python and PySpark or TypeScript Experience working with AWS services such as S3, Lambda, Glue, and related data services Solid understanding of data pipeline architectures, ETL processes … problem‐solving skills with the ability to communicate clearly with both technical and non‐technical stakeholders Preferred Qualifications Additional experience with Foundry, AIP, and PySpark Knowledge of monitoring, alerting, and logging tools and frameworks Exposure to TypeScript/React for Workshop applications and workflow orchestration Experience working within ...

Your Responsibilities: Design and maintain scalable data pipelines using PySpark, Python, and distributed computing frameworks to support high‐volume data processing. Architect and optimize AWS-based data and AI infrastructure, ensuring secure, performant, and cost‐efficient ingestion, transformation, and storage. Develop, finetune, benchmark, and evaluate GenAI/LLM models … cross-functional teams to integrate GenAI capabilities into production systems and ensure high-quality data, governance, and operational reliability Your Profile: Strong experience with PySpark, distributed data processing, and largescale ETL/ELT pipelines. Strong SQL expertise including star/snowflake schema design, indexing strategies, writing optimized queries ...

operational efficiency Write high-quality, production-ready Python code and contribute to shared analytics codebases Query and manipulate large datasets using SQL and PySpark in distributed data environments Communicate analytical insights clearly to both technical and non-technical stakeholders Ensure strong governance, risk, and control standards are embedded … experience includes: Strong background in statistical analysis and modelling Advanced programming skills in Python Experience working with large datasets using SQL and/or PySpark Highly desirable skills include: Advanced knowledge of machine learning and AI techniques Experience within financial services or strong understanding of banking products, risk frameworks ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we’re looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

models within a Lakehouse environment Own dimensional and semantic modelling to enable consistent, trusted reporting across the business Build and maintain complex SQL and PySpark transformation pipelines using Databricks Drive data quality, testing, reliability and performance across the curated or gold data layer Work closely with data engineers … experience: Strong commercial experience as an Analytics Engineer within a modern data platform or Lakehouse architecture Expert level SQL and strong hands on PySpark capability Proven experience designing and owning dimensional data models for analytics and reporting Experience working with Databricks, including pipelines and governed data environments Solid understanding ...

Lakehouse architecture Design and deliver high-quality, analytics-ready data models that support trusted reporting Translate complex business requirements into robust SQL and PySpark transformation pipelines Shape how data is structured, modelled and served across multiple domains Embed data quality, testing, reliability and performance standards into pipelines Contribute … Experience Strong commercial experience as an Analytics Engineer operating in a modern data platform Advanced data modelling capability, including dimensional and semantic modelling Strong PySpark development experience within a Lakehouse environment Advanced SQL skills for complex transformations Hands-on experience with Databricks, including pipelines and governance tooling A strong ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...

calculations, including netting sets, supervisory factors and collateral/CSA logic.* **Design a modern Databricks architecture** using Delta Lake, Unity Catalog and Spark/PySpark to support versioned, reproducible and regulator‐defensible exposure calculations.* **Develop high‐performance compute pipelines** that support intraday recalculation, scenario testing and rapid exposure analytics … exposure methodology, netting sets, supervisory parameters and collateral/CSA design.* **Strong Databricks engineering capability**, including Delta Lake versioning, Unity Catalog governance, Spark/PySpark optimisation and secure data architecture.* **Experience designing governed, audit‐ready systems**, with a focus on lineage, reproducibility, evidence trails and internal/external audit ...