1 to 25 of 33 PySpark Jobs in Manchester

practical data engineering solutions * Excellent analytical and problem-solving skills Desirable experience/qualifications will include * Azure Data Factory * Microsoft Purview * Python, Spark or PySpark * DevOps, CI/CD pipelines and Infrastructure as Code * DataOps practices * AI, machine learning or advanced analytics platforms * Dynamics 365, CRM or ERP integrations ...

deep Azure Synapse/Databricks experience with clear ability to specialise quickly in Fabric. Expert SQL/T-SQL plus strong Python or PySpark, with a track record of building maintainable ELT/ETL pipelines and analytical data models. Strong Power BI and DAX experience, including semantic modelling, incremental ...

enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform implementations Proven ability ...

such as Claude Code or equivalent tools, to improve engineering productivity while maintaining high standards of quality, governance and security. Desirable experience Regulated environment. PySpark for large-scale data transformations. Migrating legacy data estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven systems. Data-quality ...

such as Claude Code or equivalent tools, to improve engineering productivity while maintaining high standards of quality, governance and security. Desirable experience Regulated environment. PySpark for large-scale data transformations. Migrating legacy data estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven systems. Data-quality ...

Data Lakes, Blob Storage, Lakehouses and Data Modelling concepts (Kimball, star schemas) Strong ability to write Python code, including experience with data manipulation libraries (PySpark, Pandas etc) Strong ability to write SQL queries for data retrieval and manipulation Strong problem-solving skills to address and resolve data-related issues. ...

accelerate development across code, infrastructure, testing and documentation while maintaining appropriate quality and security controls. Desirable experience Financial services or another regulated environment. PySpark for large-scale batch or streaming transformations. Migrating SQL Server and SSIS estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven ...

API Gateway, and CloudWatch. Design, develop, and maintain data pipelines and ETL workflows using technologies such as Glue, Spark/PySpark, Kafka, or similar platforms. Implement and maintain Infrastructure as Code (IaC) using Terraform or equivalent tools. Drive observability, monitoring, logging, and operational readiness across software platforms and services. ...

platform or data warehouse migration (cutover, reconciliation, parallel running, decommissioning) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

come from employment, education, an apprenticeship or personal projects. Experience with Microsoft Fabric or another cloud data platform. Familiarity with data warehouses, Lakehouses , PySpark or Delta Lake. Awareness of testing and CI/CD practices. Candidates do not need to meet every desirable criterion to apply. We are proud ...

responsibilities. Ensure solutions are secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines … duties as required. Skills and Experience Strong understanding of data engineering principles, data architecture, and cloud-based data platforms. Proficient in Python (PySpark) and SQL for building and optimising data transformations at scale. Strong hands-on Databricks experience. Experience with Azure data storage technologies (Azure SQL, Data Lakes, Blob ...

looking for: Previous experience as a Technical Lead (hands-on) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

party data providers, delivering reliable and auditable data at scale. Key Responsibilities Build and maintain ingestion and transformation pipelines on Microsoft Fabric (Data Pipelines, PySpark notebooks, Dataflows Gen2) across the Landing, Curate and Enrich layers of the medallion architecture. Onboard new source systems by setting up the team … Snowflake and SQL Server, change data capture, file and API loads). Do this through configuration rather than one-off code. Write and tune PySpark and T-SQL for Lakehouse and Warehouse workloads, including fact and dimension models in the enterprise data warehouse. Implement data quality rules and validation ...

Arrow Global Group in Manchester seeks an experienced Data Engineer to design and implement robust data pipelines using Microsoft Fabric, PySpark, and SQL across multiple data sources in a financial services setting. The role focuses on building scalable data ingestion, transformation, and validation workflows, while collaborating with Data Stewards ...

Strong, demonstrable experience running A/B testing and causal inference in a product-focused environment. Core Technical Stack: Proficiency with SQL , Python/PySpark , and Snowflake . Experience with data visualisation tools (e.g. Tableau, Power BI) Hands-on experience with A/B testing and experimentation Experience with ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query ...

data ecosystem. You will lead a small team, mentor junior engineers, and collaborate with stakeholders to deliver scalable data pipelines on Databricks using PySpark and Spark SQL. The role involves guiding data strategy, migrating legacy processes, and delivering enterprise data solutions in a cloud-native environment, with hybrid #J ...

modelling, cutover approach, reconciliation strategy and technical risk management. Design and build the real‐time data architecture using Amazon MSK/Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly‐once semantics and stream‐to‐lakehouse integration. Define … Lambda, AWS S3, AWS Redshift, and Amazon EMR, with the ability to make and defend architectural decisions. Advanced proficiency in Python and SQL, including PySpark for large‐scale batch and streaming processing, and strong command of data modelling and data warehousing concepts. Production experience designing and operating streaming architectures ...

production ML systems. You will collaborate with data scientists and ML engineers, mentor junior engineers, and elevate engineering practices. The role emphasises Python, PySpark, and platforms like Databricks, Snowflake, or AWS. You will design and maintain data pipelines, ensure reproducible, point-in-time datasets, and support model versioning ...

seeking an experienced Data Engineer lead in Manchester to drive the next generation of its data estate. You will mentor junior engineers, deliver scalable PySpark pipelines on Databricks, and collaborate with product owners to shape data roadmaps. The role blends hands-on data engineering with technical leadership, governance practices ...

insight, advanced analytics and AI/ML at enterprise scale.* Develop and optimise batch and streaming ETL/ELT pipelines using Azure Data Factory, PySpark, SQL, Apache Spark and Airflow, integrated with modern tooling like dbt and Git-based CI/CD.* Implement data cataloguing, lineage tracking and governance … engineering, including designing and delivering data lake or lakehouse architectures in an enterprise cloud environment, ideally Microsoft Azure.* Hands-on expertise with Databricks, Spark, PySpark, SQL and core Azure Data Services (Data Lake Storage, Synapse Analytics, Data Factory).* Experience with data governance and cataloguing tools such as Collibra ...

Function Data Analytics/Warehousing & Business Intelligence Experience 5-10 years Location United Kingdom - Edinburgh, London, Manchester Skills Python & SQL: Advanced Python (PySpark GlobalLogic is seeking a Data & AI Architect working in our UK & Ireland Go to Market Team to drive pipeline growth and lead enterprise transformation pursuits across … mindset, with experience working across geographies including India, East Europe and North America. Proven track record of acceleration of Business growth Deep expertise in PySpark/Python, Advanced SQL, data modeling, batch/real-time streaming, and CI/CD pipelines (Git, Terraform, Databricks Asset Bundles). Deep knowledge ...

Market Risk, CCR, PnL, Trade Lifecycle and Reconciliations \n Knowledge of derivatives, swaps, futures, options and fixed-income products \n \n Desirable \n \n PySpark or distributed data processing \n Exposure to quantitative models and pricing systems \n Global banking environment experience \n 9-15 years' relevant experience ...

will need for success****Technical knowledge and experience:** * Proven experience using Databricks (essential)* Strong programming skills in Python (essential)* Experience working with: + PySpark + Large-scale data processing environments + Complex behavioural datasets* Experience cleaning, preparing and processing: + Large datasets + Real-world data containing inconsistencies ...