76 to 100 of 104 Permanent PySpark Jobs

requirements into technical solutions and actionable data initiatives Provide hands-on technical leadership across a Microsoft data stack, leveraging tools such as Databricks and PySpark Oversee end-to-end delivery of data solutions, ensuring designs are effectively implemented by engineering teams If this role sounds of interest ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

modernisation of existing data estates. 🔧 Key Responsibilities Design and build scalable data pipelines using Databricks Develop and optimise ETL/ELT workflows using Spark (PySpark/SQL) Support migration and modernisation of legacy data platforms into Databricks Improve performance, reliability and cost efficiency across data workloads Work closely with … best practices across delivery ✅ What We’re Looking For Strong hands-on experience with Databricks in a Data Engineering capacity Solid experience with Spark (PySpark/Spark SQL) Proven track record building and optimising data pipelines (ETL/ELT) Experience working within cloud environments (AWS or Azure) Strong ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

working with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

We’re looking for a highly skilled Lead Data Engineer to provide technical leadership across data engineering, delivery execution and project governance. This role ensures development standards are upheld while delivering high‐quality, reliable solutions ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

Skills and Experience Essential Skills Strong hands‐on experience with Palantir Foundry , including Code Repositories, Workshop, AIP Logic, and Automate Proficiency in Python and PySpark or TypeScript Experience working with AWS services such as S3, Lambda, Glue, and related data services Solid understanding of data pipeline architectures, ETL processes … problem‐solving skills with the ability to communicate clearly with both technical and non‐technical stakeholders Preferred Qualifications Additional experience with Foundry, AIP, and PySpark Knowledge of monitoring, alerting, and logging tools and frameworks Exposure to TypeScript/React for Workshop applications and workflow orchestration Experience working within ...

reduce false positives/negatives, and support downstream fraud and KYC models. Build and maintain scalable data pipelines and feature stores in Spark/PySpark (or Scala), including data normalization, deduplication, and feature computation across large PII datasets in AWS/Databricks environments. Lead A/B tests … proficiency in Python (preferred) or Scala, including experience with ML libraries such as scikit‐learn, XGBoost, TensorFlow or PyTorch. Extensive experience with Spark or PySpark and distributed data systems (e.g., AWS EMR, Databricks) working on very large, messy datasets. Deep understanding of supervised and unsupervised learning, feature engineering, model ...

Deliver validated AI components for abstraction and note generation. Meaningfully reduce manual QA workload through HITL optimization. Standardize testing and documentation frameworks. Establish efficient PySpark/SQL/Postgres data manipulation patterns that accelerate iteration. 3) Product & AI Context In Flight Work LLM powered abstraction and clinical note generation … Deployment owned by Platform Engineering, but AI Engineers write scalable, integration ready code. Data Gravity Data engineering is a major part of the role: PySpark, SQL/Postgres, query optimization, cloud data tooling. 4) Ideal Candidate Profile Background MSc/PhD in CS, EE, Applied Math, Stats, Physics ...

disciplinary engineering team. What You'll Be Doing as a Lead Architect Leading end-to-end delivery of a large-scale Azure, Databricks and PySpark programme, owning technical strategy across architecture, data engineering, software and QA workstreams. Acting as senior technical authority and escalation point for complex architecture … challenges into concise, business-focused language for executive decision-making. Qualifications Active UK SC clearance (essential). Desirable technical strengths: Terraform (IaC), Python and PySpark, Azure DevOps pipelines, Docker/Docker-Compose, Trivy scanning, medallion architecture with ADLS, parquet and delta tables, and strong SQL. Why Join? High-autonomy ...

build practical workflows and applications, and turn technical ideas into tools that genuinely get used. What you’ll be doing: Working across Python, SQL, PySpark and modern data tooling Building data pipelines, workflows and user-facing applications Working directly with clients to understand problems and shape solutions Translating complex … looking for: Strong experience in data engineering, software engineering or technical consulting Good hands-on ability with Python, SQL and data workflows Exposure to PySpark, Spark, ETL, data platforms or application development Comfortable working with clients, users and non-technical stakeholders SC clearance, active. Experience in defence, national security ...

experienced delivery team in a fast-paced consulting environment. 🔑 The Role Migrate existing data pipelines and tables into Databricks Build and optimise scalable PySpark-based data workflows Support Lakehouse and Delta architecture implementation Work across ETL orchestration, transformation and optimisation Collaborate directly with technical stakeholders and delivery teams 🎯 Required … must be Databricks Partner Programme vetted to be considered for the role Experience within a Databricks Delivery Partner/consulting environment Databricks Certified Strong PySpark/Spark/SQL capability Experience with ETL and data migration projects Exposure to orchestration tooling such as AWS Glue or Azure Data Factory ...

Lakehouse architecture Design and deliver high-quality, analytics-ready data models that support trusted reporting Translate complex business requirements into robust SQL and PySpark transformation pipelines Shape how data is structured, modelled and served across multiple domains Embed data quality, testing, reliability and performance standards into pipelines Contribute … Experience Strong commercial experience as an Analytics Engineer operating in a modern data platform Advanced data modelling capability, including dimensional and semantic modelling Strong PySpark development experience within a Lakehouse environment Advanced SQL skills for complex transformations Hands-on experience with Databricks, including pipelines and governance tooling A strong ...

Responsibilities Build and maintain scalable data pipelines (full ETL lifecycle) Perform data cleaning, transformation, and integration Work with Microsoft Fabric, Notebooks, Python, and PySpark Manage and develop solutions using SQL Server and Cosmos DB Handle multiple data formats and support data processing workflows Collaborate closely with Data Engineering … Business Intelligence teams across onshore and offshore locations Requirements Proven experience as a Data Engineer Strong skills in Python, PySpark, and Microsoft Fabric Experience with SQL Server and Cosmos DB Ability to design and deliver ETL pipelines Comfortable working in high-pressure environments Desirable Insurance industry experience (ideally 2+ ...

operational efficiency Write high-quality, production-ready Python code and contribute to shared analytics codebases Query and manipulate large datasets using SQL and PySpark in distributed data environments Communicate analytical insights clearly to both technical and non-technical stakeholders Ensure strong governance, risk, and control standards are embedded … experience includes: Strong background in statistical analysis and modelling Advanced programming skills in Python Experience working with large datasets using SQL and/or PySpark Highly desirable skills include: Advanced knowledge of machine learning and AI techniques Experience within financial services or strong understanding of banking products, risk frameworks ...

e.g. Data Science, Mathematics, Statistics, Physics, Computer Science, Informatics or Engineering You have strong experience with analytics and data manipulation software e.g. R, Python, PySpark, SAS, SQL, SPSS You have a strong understanding of financial markets and instruments You take a consultative approach and have polished communication and stakeholder … Diverse and inclusive environment Health and wellbeing support Volunteering opportunities Pension Apply now to find out more about this Senior Data Scientist (R Python PySpark SAS SQL SPSS) opportunity. At Client Server we believe in a diverse workplace that allows people to play to their strengths and continually learn. ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we’re looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

calculations, including netting sets, supervisory factors and collateral/CSA logic.* **Design a modern Databricks architecture** using Delta Lake, Unity Catalog and Spark/PySpark to support versioned, reproducible and regulator‐defensible exposure calculations.* **Develop high‐performance compute pipelines** that support intraday recalculation, scenario testing and rapid exposure analytics … exposure methodology, netting sets, supervisory parameters and collateral/CSA design.* **Strong Databricks engineering capability**, including Delta Lake versioning, Unity Catalog governance, Spark/PySpark optimisation and secure data architecture.* **Experience designing governed, audit‐ready systems**, with a focus on lineage, reproducibility, evidence trails and internal/external audit ...

making at scale. What You'll Be Doing: Building curated, analytics-ready datasets using modern modelling techniques Developing robust transformation pipelines using SQL and PySpark Collaborating with Data Engineers to enhance the data platform Translating business needs into scalable, well-structured data models Embedding data quality, testing, and governance … Looking For: Proven experience as an Analytics Engineer or similar role Strong SQL skills and experience with complex data transformations Hands-on experience with PySpark and modern data platforms (Lakehouse preferred) Solid understanding of dimensional modelling Strong communication skills - able to work across technical and business teams A passion ...

data models within a Lakehouse architecture Develop analytics‐ready datasets that power trusted reporting and dashboards Translate business requirements into well‐structured SQL and PySpark transformations Contribute to data modelling decisions across multiple business domains Apply data quality checks, testing and performance best practices to pipelines Work within existing … Analytics Engineer or in a similar data-focused role Solid data modelling skills, including experience with dimensional models Hands‐on experience writing PySpark in a Lakehouse or big data environment Strong SQL skills for building and maintaining transformation logic Experience working with Databricks or similar modern data platforms ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...