51 to 60 of 60 Remote/Hybrid PySpark Jobs

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

focus on designing scalable data platforms and integrating Generative AI capabilities into enterprise systems. Key Responsibilities Design, build and maintain scalable data pipelines using PySpark, Python and distributed computing frameworks Architect and optimise AWS-based data and AI infrastructure for secure, high-performance data processing Develop, fine-tune, benchmark … functional teams to integrate GenAI solutions into production environments Ensure data quality, governance and operational reliability across systems Essential Skills & Experience Strong experience with PySpark and large-scale distributed data processing (ETL/ELT pipelines) Advanced SQL expertise, including schema design (star/snowflake), indexing and optimisation techniques Proven ...

accurate documentation, lineage and operational run books to support ongoing platform support Key Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including … pipelines and code reviews What you'll need to succeed Essential Proven experience as a Data Engineer Strong hands-on experience with AzureDatabricks, Python, PySpark and SQL Experience supporting Azure data platforms in production Understanding of medallion architecture and Delta Lake Desirable Experience with Azure Synapse Analytics Experience with ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third‐party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd’s Syndicates Delegated ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

disciplinary engineering team. What You'll Be Doing as a Lead Architect Leading end-to-end delivery of a large-scale Azure, Databricks and PySpark programme, owning technical strategy across architecture, data engineering, software and QA workstreams. Acting as senior technical authority and escalation point for complex architecture … challenges into concise, business-focused language for executive decision-making. Qualifications Active UK SC clearance (essential). Desirable technical strengths: Terraform (IaC), Python and PySpark, Azure DevOps pipelines, Docker/Docker-Compose, Trivy scanning, medallion architecture with ADLS, parquet and delta tables, and strong SQL. Why Join? High-autonomy ...

delivered, and used across the business. Responsibilities Design and build end-to-end data solutions within Microsoft Fabric Develop and optimise data pipelines using PySpark and SQL Work across bronze, silver, and gold layers (medallion architecture) Perform initial data analysis and help define requirements Collaborate with analysts, Client … stakeholders Improve and automate existing processes Ensure data is secure, accurate, and scalable Requirements Technical: Strong experience with PySpark/Spark and SQL Hands-on experience with Microsoft Fabric Experience building data pipelines and models Azure Data Factory experience Understanding of modern data architectures Personal: Proactive and self-starting ...

delivered, and used across the business. Responsibilities Design and build end-to-end data solutions within Microsoft Fabric Develop and optimise data pipelines using PySpark and SQL Work across bronze, silver, and gold layers (medallion architecture) Perform initial data analysis and help define requirements Collaborate with analysts, Client … stakeholders Improve and automate existing processes Ensure data is secure, accurate, and scalable Requirements Technical: Strong experience with PySpark/Spark and SQL Hands-on experience with Microsoft Fabric Experience building data pipelines and models Azure Data Factory experience Understanding of modern data architectures Personal: Proactive and self-starting ...