1 to 25 of 70 PySpark Jobs in the UK

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

ideally in enterprise-scale environments. Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse and Azure Data Lake. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. ...

scale data platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured, semi-structured and unstructured data ...

assisted software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

such as Claude Code or equivalent tools, to improve engineering productivity while maintaining high standards of quality, governance and security. Desirable experience Regulated environment. PySpark for large-scale data transformations. Migrating legacy data estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven systems. Data-quality ...

Experience of core MS Fabric features including: OneLake, Lakehouses, Warehouses, Data Pipelines, Dataflows, Gen2, Notebooks, Eventstreams + Eventhouses. Using SQL, T-SQL, Spark SQL, PySpark + Python for data ingestion, data transformation, data optimisation, data modelling, and converting structured/unstructured source data into validatd Delta tables. Using Power ...

accelerate development across code, infrastructure, testing and documentation while maintaining appropriate quality and security controls. Desirable experience Financial services or another regulated environment. PySpark for large-scale batch or streaming transformations. Migrating SQL Server and SSIS estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven ...

challenge Technical lead or lead engineer experience on enterprise scale Databricks platforms on Azure Strong hands on data engineering background with Python or PySpark and advanced SQL Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows and Auto Loader Declarative pipelines (Delta Live Tables or Lakeflow) and Structured Streaming ...

architecture Experience with cloud data platforms (AWS preferred - Redshift, Athena, S3, Glue. Azure accepted) Understanding of workflow orchestration and automation tools Python experience (including PySpark) is beneficial but not essential Exposure to BI platforms such as Power BI, Tableau or QuickSight, with an understanding of how backend data models ...

patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data ...

patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data ...

Data modelling and data warehousing Extract-Transform-Load (ETL) Creating data pipelines Ethics in data and AI Cloud and big data technologies (AWS, Snowflake, PySpark) Working with unstructured data Working with Generative AI – Large language models (LLMs) and image generation (ChatGPT, Gemini) Consulting Skills Effective teamwork Business acumen Time ...

stakeholders to translate business challenges into analytical solutions. YOUR SKILLS AND EXPERIENCE: You will bring strong capability in: Python and SQL skills, with strong PySpark experience Hands-on expertise with Microsoft Fabric and Databricks. Experience building production-grade data pipelines, reusable data assets and machine learning solutions. Experience communicating ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

payment, and transaction analysis. Strong Python skills for statistical modelling, cohort analysis, weighting, matching, and reproducible analytics. Hands-on experience with Databricks, Spark/PySpark, Delta tables, and automated data workflows. Proficiency with Git, Azure DevOps, and controlled production release processes. Strong data quality, reconciliation, and audit-trail practices. ...

looking for: Previous experience as a Technical Lead (hands-on) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

Data Engineer Key skills: * Microsoft Fabric * Azure Cloud Technologies * Python * SQL Server/SQL * Data Warehousing & Data Modelling * Data Architecture * Data Pipelines & Integration Solutions * PySpark * ETL/ELT Development * Data Governance & Security * Source Control (Git) * Technical Leadership & Team Management You will be a Lead Data Engineer Developer with good ...

across systems Proven skills in project management, solution delivery, and budget oversight within an IT/Cloud environment Technical proficiency with SQL, Python/PySpark, and Microsoft Power Platform Experience with CI/CD, DevOps practices, and system security considerations Ability to analyse, troubleshoot, and validate data solutions, ensuring ...

making. Key Responsibilities: Lead the design and development of curated, analytics-ready dimensional and semantic data models. Build, maintain and optimise complex SQL and PySpark transformation pipelines within Databricks. Define and maintain KPI definitions, conformed dimensions and consistent business logic across data domains. Ensure data quality, integrity, testing, observability ...

best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs Use of Jupyter notebooks and/or Amazon Athena to query and validate data Data analysis ...

Strong, demonstrable experience running A/B testing and causal inference in a product-focused environment. Core Technical Stack: Proficiency with SQL , Python/PySpark , and Snowflake . Experience with data visualisation tools (e.g. Tableau, Power BI) Hands-on experience with A/B testing and experimentation Experience with ...

Platforms: Google Cloud Platform, Amazon Web Services, Microsoft Azure, and Databricks. Experience in one or more of the listed Languages such as: Python, R, PySpark Experience in one or more of the following Agentic & Gen AI Platforms: such as Microsoft Copilot Studio, Adept AI, UiPath, OpenAI GPT-5 Agents ...

data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers … latest data engineering technologies and best practices. Key Responsibilities Data Engineering & Development Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark Build and optimise ETL/ELT solutions processing large-scale datasets Develop high-quality, reusable code and engineering frameworks Deliver robust, performant and maintainable data ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query ...