26 to 50 of 66 PySpark Jobs in the UK excluding London

TensorFlow Serving, TorchServe). Familiarity with ML lifecycle tools such as MLflow, Kubeflow, or Azure ML Pipelines. Experience working with distributed data processing (e.g., PySpark) and SQL. Understanding of software engineering best practices, including version control (Git). Knowledge of CI/CD principles in ML environments. Experience with ...

commercial or Agile environment.To be successful in this role it’s essential that you will: Have significant experience of AWS , Python, SQL, Git and PySpark Desirable experience needed will be: SISS or SAS experience Quality Assurance and Test Automation experience Experience of Database technologies Experience in Financial Services organisation ...

commercial or Agile environment. To be successful in this role its essential that you will: Have significant experience of AWS , Python, SQL, Gitand PySpark Desirable experience needed will be: SISS or SAS experience Quality Assurance and Test Automation experience Experience of Database technologies Experience in Financial Services organisation About ...

migration initiatives, including consolidation of legacy data, medallion architecture design, schema evolution, and cutover planning Provide hands-on data engineering expertise, building and optimising PySpark pipelines Ensure data is secure, high-quality, accessible, and structured for maximum business value Drive best practice in data governance, cataloguing, lineage, and standards … reporting teams Ensure compliance with data protection regulations and internal policies What you'll bring Strong hands-on experience in data engineering, including PySpark and production data pipelines Experience with Microsoft Fabric, Azure Synapse, or similar platforms (e.g. Databricks, AWS) Proven ability to design and manage scalable data pipelines ...

covered) Salary: £45,000 – £95,000 - dependent on level/experience (mid, senior, principle) Security Clearance: SC active Requirements: Databricks and Spark/PySpark (absolute must haves) Azure and AWS experience Data warehouses/lakehouse platforms CI/CD and modern engineering practices Public sector/Central government ...

Strong experience designing modern cloud-based data platforms. Hands-on architectural experience with: AWS (essential): S3, EMR, Glue Kafka/event streaming architectures Python & PySpark-based data processing Experience designing data ingestion pipelines (batch and real-time). Proficiency in Infrastructure as Code (Terraform). Experience with GitHub-based ...

Strong experience designing modern cloud-based data platforms. Hands-on architectural experience with: AWS (essential): S3, EMR, Glue Kafka/event streaming architectures Python & PySpark-based data processing Experience designing data ingestion pipelines (batch and real-time). Proficiency in Infrastructure as Code (Terraform). Experience with GitHub-based ...

joins, transformations, stored procedures, and performance basics 3+ years Azure Data Factory, including pipeline development, parameter usage, and basic templates 3+ years Databricks experience (PySpark, Delta Lake, notebooks) 2+ years Python Experience Experience working with data warehouses and structured modelling concepts Strong understanding of ETL/ELT practices … data centric engineering Desirable Exposure to metadata driven or config driven engineering practices Basic PySpark optimisation or Delta Lake performance tuning Experience using Azure DevOps (repos, pipelines) Understanding of data governance, data quality tooling, or lineage concepts Experience in regulated environments or financial services The Application Timeline A first ...

support reporting, analytics, and business operations. This includes developing ETL/ELT workflows using both low-code tools and code-based approaches (SQL, Python, PySpark), ensuring performance, scalability, and reliability across the data platform. You will take ownership of data modelling and architecture , designing scalable data warehouse and lakehouse … understanding of data modelling (star schema, dimensional modelling, lakehouse design) Experience building and optimising ETL/ELT pipelines using SQL, Python, and/or PySpark Knowledge of data governance, quality, and security best practices Experience implementing CI/CD pipelines and DevOps practices Ability to work independently and manage ...

engineering within distributed systems Proven ability to build and optimise data pipelines at scale Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop Strong understanding of data ingestion and batch processing methodologies Experience with text or unstructured data is advantageous but not essential A hands ...

growing data team What they’re looking for: Strong experience with modern data platforms (Microsoft Fabric a big plus) Solid skills in Python/PySpark and SQL Experience integrating data from APIs and SaaS platforms A problem-solver who enjoys working at the intersection of tech and business ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

frameworks Engineering Delivery Design and evolve scalable data pipelines (batch and incremental watermark-based ingestion) using: Microsoft Fabric Pipelines Azure Data Factory PySpark/Python Fabric Notebooks Implement and maintain a medallion lakehouse architecture using OneLake and Fabric Warehouses Ensure Direct Lake-ready semantic models for analytics Lead incident … Hands-on experience with Microsoft Fabric, including: Workspaces, OneLake, Fabric Warehouses, Fabric Notebooks and Data Pipelines and Direct Lake semantic models Strong Python and PySpark development capability Solid SQL skills for warehouse development and metadata layers Proven ownership of production data systems end-to-end Wider Azure data platform ...

Collaborative office environment with regular social and charity events And Much More!!! DATA ENGINEER – ESSTENTIAL SKILLS Strong experience with Databricks and Apache Spark (PySpark or Scala) Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding … submit (subject to required skills) your application to our client in conjunction with this vacancy only. KEY SKILLS Data Engineering, Databricks, Apache Spark, PySpark, SQL, Python, Kafka, AWS, Delta Lake, ETL, Data Pipelines, CI/CD, Big Data ...

strategies for performance and scalability Collaborate with data scientists, analysts and DevOps teams to deliver end-to-end solutions Contribute hands-on development using PySpark, SQL and Python Support CI/CD and DevOps practices using Git-based version control Present architectural designs to governance and review forums Essential … governance, metadata and security frameworks Experience working across enterprise-scale cloud data landscapes Strong understanding of performance optimisation and data management strategies Experience with PySpark, SQL and Python Spark Desirable Skills for the Databricks Solution Architect: Strong stakeholder management and communication skills Experience working within Agile delivery environments Experience ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … best practices in data engineering, documentation, and version control Requirements Strong hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, pipelines) Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

accurate documentation, lineage and operational run books to support ongoing platform support Key Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including … pipelines and code reviews What you'll need to succeed Essential Proven experience as a Data Engineer Strong hands-on experience with AzureDatabricks, Python, PySpark and SQL Experience supporting Azure data platforms in production Understanding of medallion architecture and Delta Lake Desirable Experience with Azure Synapse Analytics Experience with ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

high-quality, governed, and observable data platform . What You’ll Be Doing Build and maintain Azure & Databricks pipelines Develop scalable ELT processes using PySpark Implement data quality, lineage, and security controls Own CI/CD pipelines and Infrastructure as Code (Terraform) Ensure pipelines are testable, observable, and easy … best practices What We’re Looking For Strong experience with Azure Data Platform & Databricks Proven track record building scalable data pipelines Hands-on with PySpark/Spark-based processing Experience with Terraform & CI/CD pipelines Solid understanding of data governance, quality, and lineage Familiarity with Git-based workflows ...

We’re looking for a highly skilled Lead Data Engineer to provide technical leadership across data engineering, delivery execution and project governance. This role ensures development standards are upheld while delivering high‐quality, reliable solutions ...