26 to 50 of 56 Remote/Hybrid PySpark Jobs

time): Drive Python-based data processes utilizing libraries like Pandas and native PySpark. Pioneer the development, maintenance, and optimization of Spark and PySpark jobs, with an emphasis on Spark SQL operations tailored for Altria's specific requirements. Cloud and Platform Management (30% of time): Steer Azure-related operations, with ...

outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

timely manner. Key Experience, Skills and Knowledge: Proven 2 yrs+ experience as a Data Engineer or similar role, with a strong focus on PySpark, SQL, Microsoft Azure Data platforms and Power BI an advantage Proficiency in development languages suitable for intermediate-level data engineers, such as: Python/PySpark ...

Azure services including Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure Functions and Azure Databricks Develop scalable pipelines using Delta Lake, PySpark and Unity Catalog Work with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance and risk teams to deliver solutions aligned to Lloyd … Azure Databricks in large scale environments Deep knowledge of Delta Lake, medallion architecture, distributed compute and lakehouse data platforms Strong development skills in Python, PySpark and Spark SQL Experience implementing CI/CD for data platforms using Azure DevOps Knowledge of data governance, lineage, access control and secure cloud ...

using Octopus Deploy or Azure functions/DevOpS is ideal. Experience working with big datasets. Develop and maintain data processing workloads using Python and PySpark, including notebook-based development where appropriate. Support the migration and modernisation of legacy databases and ETL processes to cloud-based architectures. Strong analytical ...

analysis, model development, and data interrogation. Strong knowledge of PyTorch or similar ML frameworks. Experience handling large datasets and distributed processing with PySpark or similar is desirable . Experience of experimentation tools such as MLflow is a bonus. Comfort with deploying ML models into production environments. Familiarity with ...

analysis, model development, and data interrogation. Strong knowledge of PyTorch or similar ML frameworks. Experience handling large datasets and distributed processing with PySpark or similar is desirable . Experience of experimentation tools such as MLflow is a bonus. Comfort with deploying ML models into production environments. Familiarity with ...

engineering is done across the organisation. Skills & Experience Strong hands-on experience with Azure data platforms Advanced SQL skills Commercial experience using Databricks and PySpark Proven background building and maintaining scalable data pipelines If you're looking for a role where you can combine technical depth, ownership, and influence ...

reporting, and downstream data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data … Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure acceptable) Understanding of data lake ...

AWS Data Engineer (Innovation Group) Location: London (Hybrid – 3 days per week) Duration: 6 Months Initial Contract Rate: TBD Are you a Data Engineer who thrives on building more than just standard ETL pipelines? Our ...

Modeling, attribution, lifetime value prediction, and churn analysis. They’re looking for someone with: Strong hands-on experience in Python, SQL, R, and PySpark Proven ability in marketing analytics and causal inference Great communication skills to turn complex insights into business action The position is hybrid, based in East ...

looking for a technically strong and commercially minded data professional to lead end-to-end model development. What you’ll bring: Strong Python (including PySpark) and SQL skills for data analysis, feature engineering, and modelling Experience building and deploying end-to-end predictive models from scoping through validation ...

Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance, and Solvency II teams to meet Lloyd's market … highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing CI/CD pipelines (Azure DevOps, branching/merging, automation). Solid knowledge of data governance, lineage, access ...

demonstrable commitment to developing your knowledge and expertise Significant technical data science knowledge Desirable Criteria: Understanding of supply chain data sources Experience with Pyspark/Spark SQL Experience delivering interactive visualisations in Python and SQL Graph data experience, including with graph query languages and knowledge graphs Experience working ...

datasets optimized for Power BI. Secure: Implement rigorous data security, GDPR compliance, and client ring-fencing (Row-Level Security). Innovate: Create SQL and PySpark notebooks to prepare feature engineering datasets for AI/ML models. Collaborate: Work independently via Jira/GitHub, maintaining high CI/… What You Bring 5+ years of Data Engineering experience. Expertise: Deep knowledge of Snowflake and Microsoft Fabric . Tech Stack: Advanced SQL , Python , and PySpark (Notebooks). Modelling: Strong relational data modelling and cloud cost optimization skills. DevOps: Experience with GitHub, CI/CD, and workflow orchestration. Randstad Technologies ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

Lead PySpark Engineer (Cloud Migration) Role Type: 5-Month Contract Location: Remote (UK-Based) Experience Level: Lead/Senior (5+ years PySpark) Role Overview We are seeking a Lead PySpark Engineer to drive a large-scale data modernisation project, transitioning legacy data workflows into a high-performance … cloud environment. This is a hands-on technical role focused on converting legacy SAS code into production-ready PySpark pipelines within a complex financial services landscape. Key Responsibilities Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated ...

Data Engineering Support Engineer to help keep our data pipelines and products running smoothly. This is a hands‐on role focused on PySpark and SQL, working mostly on existing jobs rather than building greenfield products. You’ll investigate and fix bugs in our data workflows, improve reliability and performance … features, even better – but the core of the job is support, stability and quality rather than product ownership. What you’ll do Monitor existing PySpark and SQL data pipelines, jobs and scheduled workflows. Investigate production issues, identify root causes, and fix bugs in existing code and configurations. Validate data ...

translating ERP data into scalable BI models Building robust Power BI datasets, reports, and semantic models Delivering complex data transformations using Python and PySpark Writing advanced T-SQL queries and performance-optimised stored procedures Acting as a senior escalation point for BI developers and coordinating workload distribution Ensuring governance … skills across modelling, DAX, optimisation, and dataset design Strong SQL capabilities (T-SQL, performance tuning, stored procedures) Experience with Python and/or PySpark for data transformation Background coordinating or leading BI developers Experience delivering BI in enterprise, multi-region environments Excellent communication skills, analytical mindset, and ability ...