1 to 25 of 30 PySpark Jobs in the North of England

tooling Key skills Strong experience with Microsoft Fabric (Data Factory, Lakehouse, OneLake, Pipelines, SQL Warehouse) Solid data engineering background using SQL, Python, PySpark, Delta Lake E xperience building and optimising end-to-end data pipelines CI/CD experience for analytics and data platforms (Azure DevOps or GitHub) Exposure ...

technical architecture and detailed design. •Expertise in building scalable ingestion frameworks (batch/streaming) using ADF, Databricks or Fabric pipelines. •Strong programming skills in PySpark, SQL, Python. •Experience implementing medallion architecture, delta lakes, and metadata‐driven pipelines. •Hands-on experience with DQ frameworks, validation rules and reconciliation logic. •Ability ...

pipelines and working with lakehouse architectures Knowledge of tools such as Azure Data Factory and/or Fabric Data Pipelines Experience with Python or PySpark for data transformation Understanding of data modelling (dimensional modelling, medallion architecture) Strong communication skills and ability to work with non-technical stakeholders Desirable: Experience ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

modern data platform development. Deep hands-on experience with Microsoft Fabric, including Lakehouse, Data Warehouse, OneLake, Pipelines and Data Factory. Strong SQL, Spark/PySpark and data modelling capability across analytical and operational datasets. Proven experience delivering complex end-to-end data platforms in Azure or Fabric environments. Strong ...

Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

business needs Ensure data quality, governance and best practice Key Requirements Strong leadership experience (5+ engineers) Solid Data Engineering/DataOps background Python/PySpark for data pipelines AWS experience (design & deployment) Strong data modelling & warehousing knowledge Agile (Jira/Azure DevOps) experience Get in touch for more details ...

data projects Key Requirements Proven leadership of multiple engineering teams Strong Data Engineering/DataOps background Experience in large, dataheavy environments Python/PySpark for endto-end pipelines Strong AWS experience (design & deployment) Deep knowledge of data modelling, warehousing & lakehouse concepts Strong stakeholder management and communication skills Agile delivery ...

practices You have experience with Databricks, ideally also Azure You have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You're excited to join a start-up in a role that you can shape and influence You're collaborative with excellent ...

will have the following skills and experience: Deep expertise with Azure Databricks, including development, deployment, and platform-level engineering. Strong software engineering capability across Pyspark, Spark SQL and Azure Data Factory. Solid understanding of DevOps practices including CI/CD and automated testing. Experience with monitoring, logging, and alerting ...

will have the following skills and experience: Deep expertise with Azure Databricks, including development, deployment, and platform-level engineering. Strong software engineering capability across Pyspark, Spark SQL and Azure Data Factory. Solid understanding of DevOps practices including CI/CD and automated testing. Experience with monitoring, logging, and alerting ...

TensorFlow Serving, TorchServe). Familiarity with ML lifecycle tools such as MLflow, Kubeflow, or Azure ML Pipelines. Experience working with distributed data processing (e.g., PySpark) and SQL. Understanding of software engineering best practices, including version control (Git). Knowledge of CI/CD principles in ML environments. Experience with ...

Kubernetes Experience in creating production grade coding and SOLID programming principles, including test-driven development (TDD) approaches Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Experience in source-control software, e.g., GitHub Proficient at communicating results in a concise manner both verbally and written Experience in data ...

notebooks) Experience building or working within modern data warehouse architectures Ability to get up and running quickly without heavy onboarding Nice to have: Python, PySpark or Spark experience Exposure to APIs and external data integration Experience working in evolving or transforming data environments Most importantly, youll be someone ...

notebooks) Experience building or working within modern data warehouse architectures Ability to get up and running quickly without heavy onboarding Nice to have: Python, PySpark or Spark experience Exposure to APIs and external data integration Experience working in evolving or transforming data environments Most importantly, youll be someone ...

other MLOps and ML Lifecycle stacks Experience with Docker and Kubernetes Experience in creating production grade coding Experience in programming languages (eg Python, PySpark, R, SAS, SQL) Experience in GitHub Strong communication skills both verbally and written ...

dashboards, DAX, M queries and semantic modelling. Build robust, optimised SQL Server data models and queries. Work within a Databricks environment (SQL/PySpark exposure is beneficial; training provided if needed). Engage directly with business stakeholders to gather requirements, understand challenges and translate them into technical solutions. Support ...

Strong experience designing modern cloud-based data platforms. Hands-on architectural experience with: AWS (essential): S3, EMR, Glue Kafka/event streaming architectures Python & PySpark-based data processing Experience designing data ingestion pipelines (batch and real-time). Proficiency in Infrastructure as Code (Terraform). Experience with GitHub-based ...

GBMs, GAMs, Random Forests, Neural Networks and clustering techniques Knowledge of statistics and distributions commonly used in insurance Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Proficient at communicating results in a concise manner both verbally and written Behaviours: Motivated by technical excellence Driven to deliver iterative ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

spoke, lake/lakehouse) Lead implementation of batch and real-time pipelines (Kafka, APIs, SFTP) Architect solutions using: AWS (S3, EMR, Glue) PySpark/Python Support cloud migrations and platform modernisation Ensure best practices in CI/CD, Terraform, observability, and governance Handle schema evolution, data quality, and performance … experience with AWS data platforms (S3, EMR, Glue) Expertise in data lake/lakehouse architectures Experience with Kafka/event-driven architectures Strong Python & PySpark skills Experience with Terraform and CI/CD pipelines Proven track record designing large-scale data solutions Desirable Experience with Microsoft Fabric (highly preferred ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...

data models within the Lakehouse Owning the transformation from enriched to curated datasets, enabling trusted reporting and insight Developing and maintaining robust SQL and PySpark transformation pipelines in Databricks Embedding data quality, testing, reliability and performance into the curated layer Working closely with data engineers, BI teams and business … Engineer within a modern data platform Excellent data modelling capability, including dimensional and semantic modelling Advanced SQL skills and strong hands on experience with PySpark Experience working with Databricks and Lakehouse architectures A solid grounding in engineering best practices, testing and data quality Confidence mentoring others and taking ownership ...