101 to 125 of 126 PySpark Jobs

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

PySpark Engineer Lead As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift you will refactor complex procedural logic into scalable, production-ready distributed pipelines … Tier-1 financial services environment. Core Responsibilities Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark ...

PySpark Engineer Lead As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift you will refactor complex procedural logic into scalable, production-ready distributed pipelines … Tier-1 financial services environment. Core Responsibilities Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog . Collaborate closely with insurance domain teams- Underwriting, Actuarial, Delegated Authority, Bordereaux, Exposure Management, Reinsurance, Finance, Risk and Solvency II -delivering solutions aligned … Azure data services and Databricks in enterprise-scale environments. Strong knowledge of Delta Lake , Medallion architecture, distributed compute and Lakehouse engineering patterns. Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps CICD , branching/merging and automated testing. Solid understanding of data governance, lineage ...

This is an end-to-end hands-on role. What You’ll Be Doing: • Designing and building end-to-end data pipelines using Databricks (PySpark/SQL/Delta Lake) • Developing and managing robust, scalable Lakehouse architectures • Implementing Delta Live Tables (DLT) for reliable, testable, production-grade pipelines • Optimising … pipelines • Collaborating with Data Scientists, Analysts, and Platform teams to improve developer experience and platform usability Key Skills & Experience: • Strong experience building pipelines with PySpark/Spark SQL • Deep understanding of Delta Lake • Databricks Platform Expertise • Hands-on experience with: Unity Catalog (governance & security), Spark Declarative Pipelines, Databricks Workflows ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

experience to Design & Develop a ETL pipelines in aws Glue for ingesting data from sql server or other sources Hands on experience on PySpark & Python Experience of implementing change data capture pipelines using aws dms for near-real time or batch ingestion Experience of developing incremental loads Glue … experience to Design & Develop a ETL pipelines in aws Glue for ingesting data from sql server or other sources Hands on experience on PySpark & Python Experience of implementing change data capture pipelines using aws dms for near-real time or batch ingestion Experience of developing incremental loads Glue ...

security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process. Optimize Spark workloads, cluster configurations, and storage … Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline conversion, data model redesign, and production cutover. Expert-Level PySpark & Python Skills. Strong ability to design, optimize, and refactor distributed data processing workflows. Advanced SQL & Data Modelling Expertise. Experience in dimensional modeling, lakehouse architecture ...

Senior Data Engineer (Python/PySpark & SQL) London (Hybrid - 3 days in office | Monday, Tuesday, Wednesday) Up to £85,000 dependent on experience Ref: J13073 We're working with a fast-growing AI SaaS business where data sits at the core of everything they build. They're looking … working with real-world, messy data and shaping it into reliable, decision-ready datasets used across product, analytics, and machine learning. Python/PySpark and SQL are central to the role. The Role: ·You'll design and maintain data pipelines, handle edge cases, and ensure data quality ...

Description: Primary Skill : PySpark, DataBricks, Databricks Lakehouse 12+ years of overall experience in data engineering, cloud architecture, or analytics platforms. Strong expertise in AWS big-data stack and Databricks Lakehouse. Hands-on experience building enterprise-grade data platforms and multi-zone Lakehouse architectures. Deep knowledge of: o Spark optimization … Customer 360/MDM o Data quality & observability Strong understanding of zero-trust security, compliance, and enterprise governance. Strong scripting/programming in Python, PySpark, SQL. ...

value from commercial data across Foundry. Key Responsibilities: Architect end-to-end AI solutions using Foundry, including machine learning models, large-scale data pipelines (PySpark), and production-grade deployment architectures within Foundry’s ecosystem. Evaluate and select optimal ML and data technologies, frameworks, and platform components within Foundry … degree in Computer Science, Engineering, or related field. Proven experience as an AI Architect, ML Engineer, or similar role implementing solutions on Foundry, PySpark, and cloud-based big data environments. Deep understanding of AI/ML algorithms, model optimization, and enterprise-grade model deployment (preferably via Foundry). Experience ...

requiring strong hands-on expertise, proactive risk management, and excellent stakeholder communication skills. Key Responsibilities Architect and implement data solutions using Databricks on AWS, PySpark, and Python Lead cloud migration initiatives, including on-prem to AWS migrations Drive GenAI-enabled automation for data asset migration Develop scalable data models … deliver actionable business insights Design and optimize ETL pipelines (Informatica or similar tools) Build and deploy PySpark applications for large-scale data processing Work on LLM pretraining, fine-tuning, augmentation, and optimization Ensure high data quality through cleaning, transformation, and validation (Python/SQL) Automate data workflows to enhance ...

translating ERP data into scalable BI models Building robust Power BI datasets, reports, and semantic models Delivering complex data transformations using Python and PySpark Writing advanced T-SQL queries and performance-optimised stored procedures Acting as a senior escalation point for BI developers and coordinating workload distribution Ensuring governance … skills across modelling, DAX, optimisation, and dataset design Strong SQL capabilities (T-SQL, performance tuning, stored procedures) Experience with Python and/or PySpark for data transformation Background coordinating or leading BI developers Experience delivering BI in enterprise, multi-region environments Excellent communication skills, analytical mindset, and ability ...

orchestration, experiment tracking, hyper-parameter tuning, and deploying automated model retraining pipelines. Programming & Data Engineering : Proficiency in object-oriented Python, advanced dataframes (Polars/Pyspark), and data versioning (DVC). Experience designing data storage solutions and using object-oriented SQL interfaces. Cloud & DevOps : Hands-on experience with at least … experience Advanced Education : A Master's degree or PhD in a relevant field is a strong plus. Parallelisation & Performance : Experience with parallelisation frameworks like Pyspark or Ray. Advanced Cloud & Infrastructure : Familiarity with serverless deployments (e.g., Fargate, Lambdas), infrastructure automation with Terratest or Ansible. ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience Hands-on experience with Microsoft Fabric, Databricks … equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Support data integration, transformation, and performance optimisation Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience … Hands-on experience with Microsoft Fabric, Databricks, or equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling Experience designing efficient, scalable data architectures ...

batch data pipelines to support fraud detection and operational decision-making. Develop event-driven solutions using Kafka, Flink, and data transformation frameworks such as PySpark or Pandas. Write maintainable, high-quality Python or Java code with strong SQL capabilities. Build and manage cloud infrastructure with AWS and Infrastructure … drive continuous improvement. What you bring 5+ experience as a senior data engineer or lead role Strong experience with AWS, SQL, Python, and PySpark, including building cloud infrastructure using IaC (CloudFormation/Terraform). Hands-on experience with streaming frameworks such as Kafka and Flink. Familiarity with data quality ...

transform them into robust, production-grade code libraries that other teams can actually use. Build Data Pipelines: Design and deploy scalable pipelines using PySpark to handle massive datasets for experiment execution. Collaborate: Work cross-functionally with Data Scientists and Software Engineers to turn complex requirements into seamless deployments. What … Looking for: You have a degree in CS, Math, or Engineering, Data Science3 Expert with Python, SQL, and PySpark . 3+ years in a similar role with a globally recognised large brand Production Experience: You’ve built and shipped scalable systems before (ideally with ML/AI components) DevOps ...

proven background and experience of full cycle development of a corporate, Microsoft based Data Warehouse. Desired skills: Design, Build, Test, Azure, Fabric Lakehouse, Spark, PySpark We are looking to recruit a high-calibre resource, so the client is happy to consider candidates from both a contract and permanent background. … your most recent CV to be considered for telephone screening. SQL/BI/ETL/DATAWAREHOUSE/AZURE/FABRIC/SPARK/PYSPARK/LEAD/MANAGER/HOME/REMOTE/MIDLANDS/BIRMINGHAM/SQL/BI/ETL/DATAWAREHOUSE/AZURE/FABRIC ...

your most recent CV to be considered for telephone screening. SQL/BI/ETL/DATAWAREHOUSE/AZURE/FABRIC/SPARK/PYSPARK/LEAD/MANAGER/HOME/REMOTE/MIDLANDS/BIRMINGHAM/SQL/BI/ETL/DATAWAREHOUSE/AZURE/FABRIC …/SPARK/PYSPARK/LEAD/MANAGER/HOME/REMOTE/MIDLANDS/BIRMINGHAM ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we’re looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

champion best practice methods; upskill the team through coaching and code reviews. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Communicate pricing proposals and insights clearly and persuasively to stakeholders at all levels. Collaborate with Modelling, Optimisation … insurance pricing. Strong technical expertise in predictive modelling techniques (e.g. GLMs and machine learning models). Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software and optimisation tools. A quantitative degree ...

TensorFlow Serving, TorchServe, ONNX, and TensorRT. Build and manage ML pipelines using MLflow, Kubeflow, and Azure ML Pipelines. Work with large-scale data using PySpark and integrate ML solutions into production environments. Monitor and improve model performance to ensure accuracy and efficiency. Contribute to team knowledge by mentoring … serving tools (ONNX, TensorRT, TensorFlow Serving, TorchServe). Experience with ML lifecycle tools (MLflow, Kubeflow, Azure ML Pipelines). Knowledge of distributed data processing (PySpark) and software engineering principles (Git). A collaborative mindset and excellent problem-solving abilities. Experience in data cleansing, exploratory data analysis, and visualisation. ...