76 to 100 of 126 PySpark Jobs

Data Quality Reporting • Strong capabilities to manage Data Catalogues and Data Lineage • Experience within Data Warehouses, Data Lake and Data Mesh • Experience with Python, Pyspark and ETL is desirable. This is an exciting opportunity to lead and influence data governance in a well-respected Media organisation during an exciting ...

efficient and well-documented code that follows a specification and can be maintained by others. You're confident using R or Python, SQL and PySpark to process, analyse and transform data at scale. You know how to create automated data pipelines and reusable workflows that handle both structured ...

efficient and well-documented code that follows a specification and can be maintained by others. You're confident using R or Python, SQL and PySpark to process, analyse and transform data at scale. You know how to create automated data pipelines and reusable workflows that handle both structured ...

efficient and well-documented code that follows a specification and can be maintained by others. You're confident using R or Python, SQL and PySpark to process, analyse and transform data at scale. You know how to create automated data pipelines and reusable workflows that handle both structured ...

efficient and well-documented code that follows a specification and can be maintained by others. You're confident using R or Python, SQL and PySpark to process, analyse and transform data at scale. You know how to create automated data pipelines and reusable workflows that handle both structured ...

Collaborative office environment with regular social and charity events And Much More!!! DATA ENGINEER – ESSTENTIAL SKILLS Strong experience with Databricks and Apache Spark (PySpark or Scala) Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding … submit (subject to required skills) your application to our client in conjunction with this vacancy only. KEY SKILLS Data Engineering, Databricks, Apache Spark, PySpark, SQL, Python, Kafka, AWS, Delta Lake, ETL, Data Pipelines, CI/CD, Big Data ...

reporting, and downstream data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data … Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure acceptable) Understanding of data lake ...

datasets optimized for Power BI. Secure: Implement rigorous data security, GDPR compliance, and client ring-fencing (Row-Level Security). Innovate: Create SQL and PySpark notebooks to prepare feature engineering datasets for AI/ML models. Collaborate: Work independently via Jira/GitHub, maintaining high CI/… What You Bring 5+ years of Data Engineering experience. Expertise: Deep knowledge of Snowflake and Microsoft Fabric . Tech Stack: Advanced SQL , Python , and PySpark (Notebooks). Modelling: Strong relational data modelling and cloud cost optimization skills. DevOps: Experience with GitHub, CI/CD, and workflow orchestration. Randstad Technologies ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

project within the Public Sector, s previous experience within the NHS/Healthcare is essential. Requirements: Python, AWS (Serverless), Lambda, StepFunctions, DynamoDB are essential PySpark & Databricks are desirable Excellent communication, leadership & stakeholder management skills are required Contract Details: Principal Python Developer 6 Month Contract Inside IR35 Hybrid ...

their data pipelines, using Python, Spark, and Databricks Workflows to deliver new platform functionality and ensure efficient execution. Key Responsibilities Develop reliable Python and PySpark code to support data ingestion, transformation, and end‐to‐end processing. Deliver new technical features and components aligned to approved solution designs and business ...

demonstrable commitment to developing your knowledge and expertise Significant technical data science knowledge Desirable Criteria: Understanding of supply chain data sources Experience with Pyspark/Spark SQL Experience delivering interactive visualisations in Python and SQL Graph data experience, including with graph query languages and knowledge graphs Experience working ...

experience to Design & Develop a ETL pipelines in aws Glue for ingesting data from sql server or other sources Hands on experience on PySpark & Python Experience of implementing change data capture pipelines using aws dms for near-real time or batch ingestion Experience of developing click apply for full ...

high-quality outcomes. Your Responsibilities: Lead the design and implementation of scalable, secure, and cost-efficient ETL/ELT pipelines using AWS Glue, Python (PySpark), and MWAA (Airflow). Define solution architectures, data models, orchestration patterns, and CI/CD for data workflows. Own the technical roadmap, decomposition …/data engineering, with 5+ years leading delivery of production solutions in an AWS data engineering environment. Advanced hands-on expertise with: Python (including PySpark & data engineering patterns) AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs ...

Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance, and Solvency II teams to meet Lloyd's market … highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing CI/CD pipelines (Azure DevOps, branching/merging, automation). Solid knowledge of data governance, lineage, access ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

maintain scalable ETL/ELT data pipelines within Microsoft Fabric Lakehouse environments Develop and optimise data ingestion/transformation workflows Develop Python and PySpark processes for data transformation and large-scale processing Support the development and optimisation of Power BI datasets, reports and dashboards Build and maintain scalable semantic … data engineering solutions Skills and Experience Hands-on expertise with Microsoft Fabric, Lakehouse architecture and Data Pipelines Experience developing Python or PySpark solutions for large-scale data processing Understanding of ETL/ELT design patterns, data modelling and modern data platform architecture Experience developing Power BI datasets, semantic models ...

year of IT work experience and must demonstrate proficiency with technologies and tools including Python, AWS, Teradata SQL, Tableau, Birst, QuickSight, Splunk, PySpark, Docker, Bootstrap, Flask, and Golang. A Bachelor's degree (or equivalent foreign education) in Computer Science, Computer Applications, Information Technology, Information Systems, Science, Technology, Business, Engineering ...

year of IT work experience and must demonstrate proficiency with technologies and tools including Python, AWS, Teradata SQL, Tableau, Birst, QuickSight, Splunk, PySpark, Docker, Bootstrap, Flask, and Golang. A Master's degree (or equivalent foreign education) in Computer Science, Computer Applications, Information Technology, Information Systems, Science, Technology, Business, Engineering ...

GBMs, GAMs, Random Forests, Neural Networks and clustering techniques Knowledge of statistics and distributions commonly used in insurance Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Proficient at communicating results in a concise manner both verbally and written Behaviours: Motivated by technical excellence Driven to deliver iterative ...

lakes, lakehouse, distributed systems) Exposure to proof of concept (PoC) development Knowledge of object storage and distributed processing frameworks Familiarity with technologies such as: PySpark, Dremio, Apache Iceberg, Apache Airflow, S3/object storage concepts, IBM Cloud (or willingness to upskill) Experience in financial services or highly regulated environments ...