76 to 100 of 115 PySpark Jobs

properly, you’ll fit right in. You’ll Work With Azure Data Factory, Data Lake, Key Vault, Azure Functions Databricks, Delta Lake, Unity Catalog, PySpark, Spark SQL Medallion architecture, Lakehouse patterns, distributed compute CI/CD pipelines using Azure DevOps Cross‐functional teams in underwriting, actuarial, delegated authority, exposure … have: Strong hands‐on experience with Azure Data Services and Databricks Solid understanding of Delta Lake, Medallion architecture and Lakehouse patterns Proficiency in Python, PySpark and Spark SQL Experience building CI/CD pipelines using Azure DevOps Knowledge of data governance, lineage, access controls and FinOps Confident communication skills ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … best practices in data engineering, documentation, and version control Requirements Strong hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, pipelines) Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

accurate documentation, lineage and operational run books to support ongoing platform support Key Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including … pipelines and code reviews What you'll need to succeed Essential Proven experience as a Data Engineer Strong hands-on experience with AzureDatabricks, Python, PySpark and SQL Experience supporting Azure data platforms in production Understanding of medallion architecture and Delta Lake Desirable Experience with Azure Synapse Analytics Experience with ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

Role Overview The role is responsible for designing, building & maintaining robust data architectures that integrate SAP and non-SAP systems to support enterprise-wide data management, analytics, and reporting needs. This role ensures seamless data ...

requirements into technical solutions and actionable data initiatives Provide hands-on technical leadership across a Microsoft data stack, leveraging tools such as Databricks and PySpark Oversee end-to-end delivery of data solutions, ensuring designs are effectively implemented by engineering teams If this role sounds of interest ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … Leverage AI-assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric & Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack What's on Offer: Salary: £70,000 Excellent benefits & annual leave package Strong progression & development ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

modernisation of existing data estates. 🔧 Key Responsibilities Design and build scalable data pipelines using Databricks Develop and optimise ETL/ELT workflows using Spark (PySpark/SQL) Support migration and modernisation of legacy data platforms into Databricks Improve performance, reliability and cost efficiency across data workloads Work closely with … best practices across delivery ✅ What We’re Looking For Strong hands-on experience with Databricks in a Data Engineering capacity Solid experience with Spark (PySpark/Spark SQL) Proven track record building and optimising data pipelines (ETL/ELT) Experience working within cloud environments (AWS or Azure) Strong ...

optimise end-to-end data pipelines Act as Databricks SME partnering with stakeholders to gather requirements and drive continuous improvement Develop solutions using PySpark, Spark SQL and Python Implement and maintain CI/CD practices Enhance and optimise existing data pipelines About You Proven experience in Databricks, Delta Lake … PySpark and Spark SQL Excellent communication skills - ability to partner with the business and key stakeholders to champion best practice in data engineering Strong Python and SQL skills Knowledge of CI/CD practices On Offer Salary - up to £75,000 Bonus - up to 15% 27 days annual leave ...

working with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

We’re looking for a highly skilled Lead Data Engineer to provide technical leadership across data engineering, delivery execution and project governance. This role ensures development standards are upheld while delivering high‐quality, reliable solutions ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

ETL Data Engineer/Data Architect - Azure Stack We are looking for an experienced ETL Data Engineer/Data Architect to deliver projects in collaboration with our IT partners and internal stakeholders that will transform ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

Skills and Experience Essential Skills Strong hands‐on experience with Palantir Foundry , including Code Repositories, Workshop, AIP Logic, and Automate Proficiency in Python and PySpark or TypeScript Experience working with AWS services such as S3, Lambda, Glue, and related data services Solid understanding of data pipeline architectures, ETL processes … problem‐solving skills with the ability to communicate clearly with both technical and non‐technical stakeholders Preferred Qualifications Additional experience with Foundry, AIP, and PySpark Knowledge of monitoring, alerting, and logging tools and frameworks Exposure to TypeScript/React for Workshop applications and workflow orchestration Experience working within ...

reduce false positives/negatives, and support downstream fraud and KYC models. Build and maintain scalable data pipelines and feature stores in Spark/PySpark (or Scala), including data normalization, deduplication, and feature computation across large PII datasets in AWS/Databricks environments. Lead A/B tests … proficiency in Python (preferred) or Scala, including experience with ML libraries such as scikit‐learn, XGBoost, TensorFlow or PyTorch. Extensive experience with Spark or PySpark and distributed data systems (e.g., AWS EMR, Databricks) working on very large, messy datasets. Deep understanding of supervised and unsupervised learning, feature engineering, model ...

Deliver validated AI components for abstraction and note generation. Meaningfully reduce manual QA workload through HITL optimization. Standardize testing and documentation frameworks. Establish efficient PySpark/SQL/Postgres data manipulation patterns that accelerate iteration. 3) Product & AI Context In Flight Work LLM powered abstraction and clinical note generation … Deployment owned by Platform Engineering, but AI Engineers write scalable, integration ready code. Data Gravity Data engineering is a major part of the role: PySpark, SQL/Postgres, query optimization, cloud data tooling. 4) Ideal Candidate Profile Background MSc/PhD in CS, EE, Applied Math, Stats, Physics ...

disciplinary engineering team. What You'll Be Doing as a Lead Architect Leading end-to-end delivery of a large-scale Azure, Databricks and PySpark programme, owning technical strategy across architecture, data engineering, software and QA workstreams. Acting as senior technical authority and escalation point for complex architecture … challenges into concise, business-focused language for executive decision-making. Qualifications Active UK SC clearance (essential). Desirable technical strengths: Terraform (IaC), Python and PySpark, Azure DevOps pipelines, Docker/Docker-Compose, Trivy scanning, medallion architecture with ADLS, parquet and delta tables, and strong SQL. Why Join? High-autonomy ...