1 to 25 of 33 PySpark Jobs in the North of England

technical architecture and detailed design. •Expertise in building scalable ingestion frameworks (batch/streaming) using ADF, Databricks or Fabric pipelines. •Strong programming skills in PySpark, SQL, Python. •Experience implementing medallion architecture, delta lakes, and metadata‐driven pipelines. •Hands-on experience with DQ frameworks, validation rules and reconciliation logic. •Ability ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

modern data platform development. Deep hands-on experience with Microsoft Fabric, including Lakehouse, Data Warehouse, OneLake, Pipelines and Data Factory. Strong SQL, Spark/PySpark and data modelling capability across analytical and operational datasets. Proven experience delivering complex end-to-end data platforms in Azure or Fabric environments. Strong ...

code, with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … architectural decisions, and help mentor junior engineers as the team scales. Responsibilities Architect and write production-grade ELT/ETL data pipelines using PySpark and Python within Azure ecosystem. Build custom, reusable data processing frameworks and libraries in Python/Scala to streamline ingestion and transformation tasks across ...

Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

business needs Ensure data quality, governance and best practice Key Requirements Strong leadership experience (5+ engineers) Solid Data Engineering/DataOps background Python/PySpark for data pipelines AWS experience (design & deployment) Strong data modelling & warehousing knowledge Agile (Jira/Azure DevOps) experience Get in touch for more details ...

Customer Experience, Finance and Operations can trust. Responsibilities Build, optimise and operate ELT/ETL pipelines into our data platform using SQL and Python (PySpark), with a focus on reliability, performance and maintainability. Develop and maintain core data models (curated layers, dimensional models, shared definitions) that enable consistent ...

Support production systems and troubleshoot issues Write clean, tested, and maintainable code Tech Stack Python & SQL AWS (EMR, Athena, Lambda) Apache Spark (Scala or PySpark) Big data/distributed processing tools What They’re Looking For Strong experience as a Data Engineer using Python and SQL Experience working with ...

sizing Fabric initiatives, including capacity needs, effort estimates, infrastructure requirements, team resourcing, and delivery timelines. Extensive knowledge in data modelling, DAX, Python/PySpark, SQL/KQL. Proven record of technical leadership and effective stakeholder engagement. Lead architectural design, capacity planning, and performance optimisation efforts. Implement governance, security ...

data projects Key Requirements Proven leadership of multiple engineering teams Strong Data Engineering/DataOps background Experience in large, dataheavy environments Python/PySpark for endto-end pipelines Strong AWS experience (design & deployment) Deep knowledge of data modelling, warehousing & lakehouse concepts Strong stakeholder management and communication skills Agile delivery ...

Collaborate with your peers on technical direction within your team Required Skills and Experience Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

design and optimisation within Microsoft Fabric Lakehouse, including Data Pipelines, Notebooks, and cloud-based ingestion/transformation layers. Deliver complex SQL, Python, and PySpark solutions for large-scale distributed processing. Coordinate workloads across BI developers and mentor junior and mid-level team members. Ensure BI quality, governance, documentation ...

design and optimisation within Microsoft Fabric Lakehouse, including Data Pipelines, Notebooks, and cloud-based ingestion/transformation layers. Deliver complex SQL, Python, and PySpark solutions for large-scale distributed processing. Coordinate workloads across BI developers and mentor junior and mid-level team members. Ensure BI quality, governance, documentation ...

production. Key skills and experience: Strong experience as a Data Engineer Advanced Python and SQL skills Experience with cloud platforms (AWS preferred) Proficiency in PySpark and/or Pandas Familiarity with data quality tools (e.g., Great Expectations) Experience with infrastructure as code (Terraform or CloudFormation) Understanding of modern development ...

will have the following skills and experience: Deep expertise with Azure Databricks, including development, deployment, and platform-level engineering. Strong software engineering capability across Pyspark, Spark SQL and Azure Data Factory. Solid understanding of DevOps practices including CI/CD and automated testing. Experience with monitoring, logging, and alerting ...

will have the following skills and experience: Deep expertise with Azure Databricks, including development, deployment, and platform-level engineering. Strong software engineering capability across Pyspark, Spark SQL and Azure Data Factory. Solid understanding of DevOps practices including CI/CD and automated testing. Experience with monitoring, logging, and alerting ...

notebooks) Experience building or working within modern data warehouse architectures Ability to get up and running quickly without heavy onboarding Nice to have: Python, PySpark or Spark experience Exposure to APIs and external data integration Experience working in evolving or transforming data environments Most importantly, youll be someone ...

notebooks) Experience building or working within modern data warehouse architectures Ability to get up and running quickly without heavy onboarding Nice to have: Python, PySpark or Spark experience Exposure to APIs and external data integration Experience working in evolving or transforming data environments Most importantly, youll be someone ...

teams. What we're looking for: Strong people leadership experience (5+ team size) Solid Data Engineering/DataOps background Experience with AWS, Python/PySpark, ETL/ELT pipelines Strong understanding of data platforms (Data Lake/Lakehouse/EDW) Hybrid working (3 days in office) + strong bonus ...

Kubernetes Experience in creating production grade coding and SOLID programming principles, including test-driven development (TDD) approaches Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Experience in source-control software, e.g., GitHub Proficient at communicating results in a concise manner both verbally and written Experience in data ...

commercial or Agile environment. To be successful in this role it's essential that you will: Have experience of AWS , Python, SQL, Gitand PySpark Desirable experience needed will be: SISS or SAS experience Quality Assurance and Test Automation experience Experience of Database technologies Experience in Financial Services organisation About ...

data for critical client projects. How to develop and enhance your knowledge of agile ways of working and working in open source stack (PySpark/PySql). Quality engineering professionals utilise Accenture delivery assets to plan and implement quality initiatives to ensure solution quality throughout delivery. As a Data ...

Data Architect - Databricks - Hybrid A leading Microsoft partnered consultancy is seeking an experienced Data Architect to design and deliver modern data solutions built on Databricks Lakehouse Architecture. You'll work with a variety of enterprise ...

using Octopus Deploy or Azure functions/DevOpS is ideal. Experience working with big datasets. Develop and maintain data processing workloads using Python and PySpark, including notebook-based development where appropriate. Support the migration and modernisation of legacy databases and ETL processes to cloud-based architectures. Strong analytical ...

efficient and well-documented code that follows a specification and can be maintained by others. You're confident using R or Python, SQL and PySpark to process, analyse and transform data at scale. You know how to create automated data pipelines and reusable workflows that handle both structured ...