76 to 100 of 124 PySpark Jobs in England

Collaborative office environment with regular social and charity events And Much More!!! DATA ENGINEER – ESSTENTIAL SKILLS Strong experience with Databricks and Apache Spark (PySpark or Scala) Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding … submit (subject to required skills) your application to our client in conjunction with this vacancy only. KEY SKILLS Data Engineering, Databricks, Apache Spark, PySpark, SQL, Python, Kafka, AWS, Delta Lake, ETL, Data Pipelines, CI/CD, Big Data ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

scalable, automated data pipelines across SQL databases, APIs, cloud platforms, and business systems (including Dynamics 365) Develop data transformation processes using SQL, Python/PySpark, and modern data tools Implement data quality checks, validation, and monitoring to ensure accuracy and reliability Structure and model data to support BI reporting … building automated data pipelines using cloud platforms (e.g. Microsoft Fabric, Azure Data Factory, Synapse or similar) Strong SQL skills and experience with Python or PySpark Experience integrating data from multiple systems, including APIs and business platforms Understanding of data quality, validation, and governance principles Ability to deliver well-structured ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … best practices in data engineering, documentation, and version control Requirements Strong hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, pipelines) Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

focus on designing scalable data platforms and integrating Generative AI capabilities into enterprise systems. Key Responsibilities Design, build and maintain scalable data pipelines using PySpark, Python and distributed computing frameworks Architect and optimise AWS-based data and AI infrastructure for secure, high-performance data processing Develop, fine-tune, benchmark … functional teams to integrate GenAI solutions into production environments Ensure data quality, governance and operational reliability across systems Essential Skills & Experience Strong experience with PySpark and large-scale distributed data processing (ETL/ELT pipelines) Advanced SQL expertise, including schema design (star/snowflake), indexing and optimisation techniques Proven ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

Role Overview The role is responsible for designing, building & maintaining robust data architectures that integrate SAP and non-SAP systems to support enterprise-wide data management, analytics, and reporting needs. This role ensures seamless data ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third‐party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd’s Syndicates Delegated ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

We’re looking for a highly skilled Lead Data Engineer to provide technical leadership across data engineering, delivery execution and project governance. This role ensures development standards are upheld while delivering high‐quality, reliable solutions ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

Role GenAI Data Engineer Your responsibilities: Design and maintain scalable data pipelines using PySpark, Python, and distributed computing frameworks to support high‐volume data processing. Architect and optimize AWS-based data and AI infrastructure, ensuring secure, performant, and cost‐efficient ingestion, transformation, and storage. Develop, finetune, benchmark, and evaluate … capabilities into production systems and ensure high-quality data, governance, and operational reliability Your Profile Essential skills/knowledge/experience: Strong experience with PySpark, distributed data processing, and largescale ETL/ELT pipelines. Strong SQL expertise including star/snowflake schema design, indexing strategies, writing optimized queries ...