51 to 75 of 142 PySpark Jobs in London

growth across regulated industries.ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules, logging and alerting to data pipelinesBuild enterprise semantic models (star ...

ingest, cleanse and model data for BI reporting, ML and AI, ensuring quality and governance. The role emphasizes strong Azure skills (ADF, ADLS, Databricks), PySpark, Python and SQL, with a DevOps and CI/CD mindset. Hybrid work with three days in the office weekly is offered. #J ...

investigate complex problems and take ownership of technical solutions. Strong communication skills and a collaborative approach. Desirable Experience Data engineering technologies such as Databricks, PySpark or Spark. Cloud platforms, particularly AWS. Python frameworks such as FastAPI and PyTorch. Experience developing APIs and data pipelines. Infrastructure as Code, CI/ ...

agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL.Solid understanding of data engineering, data modelling, security and governance in enterprise settings. Experience with software engineering best practices including ...

experience: Proven experience building production-grade data pipelines and ETL processes. Excellent Python skills with experience developing scalable data solutions. Experience working with PySpark or distributed data processing frameworks. Experience using GitHub and modern software engineering practices. Understanding of cloud-based data platforms, ideally Azure. Experience working with sensitive ...

Lake and Lakehouse architecture for efficient data management; experience with ETL processes and optimizing data pipelines for performance Be strong in Python, SQL, and PySpark Be passionate about building data products for stakeholders and customers, ensuring they are stable, scalable, secure, accurate, observable, and performant Enjoy collaborating closely with ...

Gen2), Purview, Cosmos DB, OpenAI, Azure ML, and AI Foundry Solid understanding of software engineering concepts, with hands‐on experience using Python, Scala or PySpark, data modelling techniques, database technologies, and SQL Confident communicator, able to explain technical concepts to non‐technical audiences and mentor junior colleagues For Senior ...

What will you need to succeed? Strong production‐level Python engineering experience, with the ability to build maintainable, tested, and supportable services. Experience with PySpark or equivalent large‐scale data processing frameworks. Hands‐on experience designing and operating CI/CD pipelines using Azure DevOps or similar tooling. Experience ...

building scalable data pipelines, Databricks lakehouse architectures, and robust data platforms in a modern cloud environment. Strong expertise in Python, Databricks, Spark/PySpark and SQL is required, with hands-on delivery across ETL/ELT, data quality and CI/CD practices. Banking domain experience is a plus ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

management, network design and access control; Databricks and OCI, or comparable platforms Strong Python and SQL, and experience building and monitoring production pipelines using PySpark or Spark Experience with infrastructure as code, CI/CD and a Git and GitHub workflow A track record of setting engineering standards ...

Data Lake Storage (Gen2), Purview, Cosmos DB, OpenAI, AzureML, AIFoundry, Kubernetes Understanding of software engineering tools and concepts including experience in Python, Scala or PySpark, database technologies, data modelling and SQL Confident communicator who is able to explain technical terms to non‐technical audiences and mentor junior colleagues Leads ...

software or computer engineering, applied math, physics, statistics, or a related field or equivalent experience Technical Skills: Strong knowledge of distributed data processing (Spark, PySpark, SQL) Expertise in data governance, metadata management, lineage, and data modeling Experience with AI/ML platforms and AI‐driven architecture Proven ability ...

/CD, monitoring, deployment patterns such as blue-green and canary) Cloud and data platforms (Azure, AWS or GCP; Databricks, Snowflake); Python, SQL, PySpark Experience leading proposals, solution architecture and commercial estimation in a consulting or services environment Confident communicator able to explain complex AI concepts to senior business ...

pipelines using modern cloud-native data engineering technologies. Provide technical direction across data modelling, ingestion, transformation, orchestration and integration. Apply practical experience of Spark, PySpark, Python and SQL within enterprise data environments. Governance, Security & Quality Embed effective data governance, security, quality and access controls within cloud data environments. Ensure … Fabric or another cloud data platform. Strong understanding of modern data architecture patterns, data modelling and ETL/ELT approaches. Practical experience with Spark, PySpark, Python and SQL and the design of scalable batch and real-time data pipelines. Experience designing appropriate data governance, security and data quality controls. ...

resolve data engineering challenges, supporting continuous platform improvement. Migrate data from legacy systems into Azure-based data lakes and data warehouses. Develop and maintain PySpark notebooks and data processing solutions. Improve data quality, reliability, and performance across data pipelines. Leverage AI-driven tools to enhance coding, testing, and documentation … data lakes, and end-to-end data pipelines. Proficiency in SQL, stored procedures, and working with relational and NoSQL databases. Hands-on experience with PySpark and Microsoft Fabric. Knowledge of DevOps, CI/CD, Agile delivery, and data quality best practices. Strong analytical, problem-solving, and troubleshooting skills. Experience ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

Location: London; UKWork type: HybridSeniority: SeniorPrimary skill: Data Software EngineeringPosted: 2026-09-29We're looking for a Senior Data Engineer – Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode. In this role, you will lead the design, development and optimization … scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. You will work hands-on to deliver performant data solutions for high-volume workloads, ensuring governance, reliability and best practices for enterprise-grade platforms. As a technical leader, you will define data strategies, drive modernization initiatives ...

experience with Databricks — running queries, navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation. Experience with Databricks and PySpark for large‐scale data processing. Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring. Experience implementing responsible ...

ODCS/ODPS), data product portals, lineage tools. Experience in a regulated financial services company. Experience leading a migration or a large platform change. PySpark or other distributed processing experience. We're building a place where everyone truly feels they belong. Even if your past experience doesn't align ...

target reconciliation, validate financial calculations, and ensure 100% data integrity across Bronze/Silver/Gold layers. Responsibilities include building automated frameworks with PyTest, PySpark, and advanced SQL, integrating checks into Azure DevOps pipelines, and monitoring schema drift and #J-18808-Ljbffr ...

data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers … latest data engineering technologies and best practices. Key Responsibilities Data Engineering & Development Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark Build and optimise ETL/ELT solutions processing large-scale datasets Develop high-quality, reusable code and engineering frameworks Deliver robust, performant and maintainable data ...

work Microsoft Fabric sits at the centre of the platform: Lakehouse, Warehouse, OneLake, pipelines and notebooks, built around a medallion architecture. Youll use Python, PySpark and advanced SQL to build the pipelines, models and integrations behind the businesss reporting, analytics and wider data needs. Youll also: Integrate internal … least a year in that role working with Fabric, so youve lived with your own design decisions. Youll also bring: Strong Python/PySpark and advanced SQL Sound data modelling, ETL/ELT, orchestration and data quality experience A good grasp of data governance, privacy and security ...

troubleshooting of Azure DatabricksDatabricks Serverless configuration and workload optimisationDatabricks SQL and Delta Lake developmentCluster management, compute policies, autoscaling, and workload monitoringData EngineeringStrong Python, PySpark, and SQL development skillsDesign and build ingestion and transformation pipelinesFull and incremental data loadingCDC (Change Data Capture) implementationData quality controls, schema evolution, reconciliation, and error ...