26 to 43 of 43 PySpark Jobs in the UK excluding London

assure enterprise-aligned data integration architectures Optimise cloud platforms (Azure, AWS, Databricks) for performance and cost Build and own ETL/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure … reviews Collaborate with stakeholders to translate complex technical decisions clearly Support strategic roadmap, tooling and continuous improvement initiatives Core Skills & Expertise Data Engineering Advanced PySpark and SQL, performance tuning and best practice coding Data modelling (conceptual, logical, physical) and integration design Data quality, testing frameworks and automated validation Strong ...

understanding of DAG (Directed Acyclic Graphs) and their role in pipeline orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic ...

Azure Data Platform and Databricks ecosystem. The ideal candidate should have strong hands-on experience with Structured Streaming, Delta Live Tables (DLT), PySpark architecture, Python, and SQL, and a proven track record of delivering large-scale enterprise data platforms. You’ll be responsible for: Leading the design, development ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

between security and broader technology teams to demonstrably improve the overall security posture. Desirable skills and experience: Scripting and automation capabilities, particularly utilising Python, PySpark, and SQL to streamline analytical workflows. Experience navigating cloud-native security environments (AWS, Azure, or GCP) alongside familiarity with the Microsoft 365 ecosystem. Advanced ...

data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions Provide technical leadership and architectural guidance across data platform initiatives … leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced SQL and distributed data systems expertise ...

migration initiatives, including consolidation of legacy data, medallion architecture design, schema evolution, and cutover planning Provide hands-on data engineering expertise, building and optimising PySpark pipelines Ensure data is secure, high-quality, accessible, and structured for maximum business value Drive best practice in data governance, cataloguing, lineage, and standards … reporting teams Ensure compliance with data protection regulations and internal policies What you'll bring Strong hands-on experience in data engineering, including PySpark and production data pipelines Experience with Microsoft Fabric, Azure Synapse, or similar platforms (e.g. Databricks, AWS) Proven ability to design and manage scalable data pipelines ...

structured and unstructured datasets Validate large-scale data ingestion, transformation, and processing workflows Ensure data integrity, transformation accuracy, reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis … DynamoDB, AWS CloudWatch, AWS IAM, Python, SQL, Java, Bash, GitLab CI/CD, Jupyter Notebooks, Apache Spark/PySpark, Terraform, Docker Key Skills Strong proficiency in Python, SQL, Java, and Scripting languages such as Bash Hands-on experience testing Big Data platforms and Data Lake solutions Strong experience with ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … Microsoft Fabric (Lakehouse, Warehouse, pipelines). Azure Data Factory experience will also be considered if no working experience with fabric Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

visible role where you'll combine strong technical skills with commercial thinking to drive meaningful product improvements. Your Skills & Experience Strong SQL skills Python (PySpark experience is beneficial) Experience in Product Analytics Strong knowledge of A/B testing and experimentation Excellent statistical analysis skills, with exposure to modelling ...

Senior Data Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data Engineer, Data Engineer) Our client is a global innovator and world leader … with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience is also key, as are excellent communication skills ...

accurate documentation, lineage and operational run books to support ongoing platform support Key Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including … pipelines and code reviews What you'll need to succeed Essential Proven experience as a Data Engineer Strong hands-on experience with AzureDatabricks, Python, PySpark and SQL Experience supporting Azure data platforms in production Understanding of medallion architecture and Delta Lake Desirable Experience with Azure Synapse Analytics Experience with ...

days holiday, pension, healthCareer Progression + Modern Tech StackPyspark | ML | AI | Databricks | Kafka | AWSAre you a Data Engineer with strong Python, SQL and PySpark skills who is looking for the opportunity to step up and accelerate your career We're partnering with a market-leading technology business that … Build and optimise large-scale data pipelines Work with billions of data points generated from IoT devices Develop solutions using Python, SQL and PySpark Gain exposure to Databricks, Kafka and streaming technologies Contribute to Machine Learning and AI initiatives Work alongside Data Scientists and Software Engineers Influence architecture ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Support data integration, transformation, and performance optimisation Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience … Hands-on experience with Microsoft Fabric, Databricks, or equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling Experience designing efficient, scalable data architectures ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we're looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

with Microsoft Fabric (Lakehouses, Pipelines, Notebooks) in a live production environment Building, enhancing Data Factory pipelines as part of an ongoing migration into Fabric PySpark for data transformation, engineering within Fabric notebooks SQL queries, transformation logic within Fabric CRM data (Dynamics 365/Dataverse), supporting integration, modelling, optimisation Business … experience Strong SQL skills, including complex queries, transformations Hands-on experience with Microsoft Fabric Experience building, maintaining data pipelines (Data Factory/Fabric Pipelines) PySpark experience for data engineering, transformation Exposure to CRM data, understanding of how CRM supports business processes Ability to work with stakeholders to identify data ...

Microsoft Fabric (Lakehouses, Pipelines, Notebooks) in a live production environment Building and enhancing Data Factory pipelines as part of the ongoing migration into Fabric PySpark for data transformation and engineering within Fabric notebooks SQL for querying and implementing transformation logic within Fabric Dynamics 365/Dataverse data, supporting integration … including complex queries and data transformations Hands-on experience with Microsoft Fabric Proven experience building and maintaining data pipelines (Data Factory/Fabric Pipelines) PySpark experience for data engineering and transformation Exposure to CRM data, with an understanding of how it underpins business processes Ability to engage with stakeholders ...

agentic AI systems using modern LLM frameworks Develop and deploy Generative AI solutions across production environments Work with structured and unstructured data using Python & PySpark Implement Claude Code/LLM-based workflows for real-time decision-making Collaborate with cross-functional teams to integrate AI capabilities into existing platforms … with Agentic AI architectures Hands-on expertise with LLMs (e.g. Claude, OpenAI, etc.) Proven background in Generative AI (GenAI) Advanced proficiency in Python and PySpark Experience working with Claude Code or similar LLM orchestration tools Strong engineering mindset with experience deploying into production environments Desirable Experience working within Financial ...