26 to 50 of 56 PySpark Jobs in London

Strong experience designing modern cloud-based data platforms. Hands-on architectural experience with: AWS (essential): S3, EMR, Glue Kafka/event streaming architectures Python & PySpark-based data processing Experience designing data ingestion pipelines (batch and real-time). Proficiency in Infrastructure as Code (Terraform). Experience with GitHub-based ...

governed data solutions using modern cloud technologies. The Role Own the end-to-end design, build, optimisation, and support of scalable Spark/PySpark data pipelines (batch and streaming) Define and implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality controls Design … solutions Required Skills & Experience Strong experience as a Senior or Lead Data Engineer with ownership of end-to-end data solutions Expertise in Databricks, PySpark/Spark, SQL, and Python Proven experience building and optimising large-scale data pipelines in production environments Strong knowledge of cloud data architectures , particularly ...

governed data solutions using modern cloud technologies. The Role Own the end-to-end design, build, optimisation, and support of scalable Spark/PySpark data pipelines (batch and streaming) Define and implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality controls Design … solutions Required Skills & Experience Strong experience as a Senior or Lead Data Engineer with ownership of end-to-end data solutions Expertise in Databricks, PySpark/Spark, SQL, and Python Proven experience building and optimising large-scale data pipelines in production environments Strong knowledge of cloud data architectures , particularly ...

Technology Environment You will be working across a modern Azure data stack, including: Azure Databricks Azure Data Factory (ADF) Synapse/Fabric Snowflake Python & PySpark SQL DevOps & CI/CD tooling within Azure Experience Required: Strong experience in modern data engineering within trading or financial services (energy/commodities … experience highly desirable) Deep hands-on experience with Azure Databricks Strong Python programming capability (including PySpark) Advanced SQL skills Experience building and orchestrating scalable data pipelines Strong understanding of data modelling and architecture principles Familiarity integrating data platforms with Power BI and/or Tableau Appreciation for data governance ...

Architecture . Build and optimize batch and real-time data pipelines across underwriting, pricing, claims, reinsurance, and bordereaux data. Develop scalable solutions using Python, PySpark, Databricks, and/or Snowflake . Integrate data from policy administration systems, claims platforms, broker systems, third-party providers, and market feeds. Ensure strong … background in Insurance , ideally within the London Market ecosystem. Deep knowledge of underwriting, claims, delegated authority, pricing, and reinsurance. Hands-on expertise in Python, PySpark, Databricks, Snowflake, SQL . Experience with Azure, AWS, or GCP . Strong understanding of batch/streaming pipelines and data modelling. Familiarity with DevOps ...

outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

direction, maintaining standards, and driving output Designing and delivering data solutions across platforms, including low/no-code tooling and code-based approaches (Python, PySpark, SQL) Overseeing the development of user-facing applications and workflows using TypeScript and React Acting as the primary client interface, shaping requirements and translating ...

processing Handson in Prompt Engineering Azure Open AI Form Recognizer Cognitive Search Vector Databases Develop and deliver upskilling sessions to the customer Python and PySpark Randstad Technologies is acting as an Employment Business in relation to this vacancy. ...

Azure or AWS Developing modern lakehouse architectures using: Databricks (Spark, SQL, Delta Lake) Azure Data Lake Storage or Amazon S3 Writing high-quality PySpark, Python, and SQL for ingestion, transformation, and modelling Implementing Bronze/Silver/Gold data layers Embedding data quality, validation and monitoring within pipelines Supporting … Data Engineering roles Strong experience working with Azure or AWS cloud platforms Commercial, hands-on experience using Databricks in production Proven capability across: PySpark/Python/SQL Cloud data storage (ADLS or S3) ETL/ELT pipeline development Solid understanding of data architecture, modelling, and governance Experience building ...

national scale. You must already hold active SC clearance to be considered. Responsibilities Design, build and maintain Databricks data pipelines using Apache Spark (PySpark & Spark SQL) Implement Delta Lake/Lakehouse architectures (Bronze, Silver, Gold) Develop scalable ELT/ETL workflows for batch and streaming data Optimise performance, reliability … delivery environment Contribute to CI/CD pipelines and automated testing practices Essential Skills & Experience Proven commercial experience as a Databricks Data Engineer Strong PySpark and SQL capability Hands-on delivery within AWS, Azure or GCP Experience with Delta Lake , data lakes and data warehouses Strong understanding of data ...

/PnL/Trade lifecycle/Reconciliation Understanding of financial products such as: Derivatives (swaps, futures, options), bonds Good-to-Have Exposure to PySpark or large-scale data processing Experience working with quants or pricing models Basic scripting (Unix/Bash) Prior experience in global banking environments Experience ...

ownership of data pipelines and automation of data workflows. Finally have 5 years Strong programming experience at an advanced level in Python, PySpark, and SQL scripting. Benefits Salary: 50,000 - 65,000 per annum 25 day's holiday Company car/Allowance Variable annual bonus based 5-15% Pension … leveraging tools such as Foundry Code Workbooks, Pipelines, and Object Explorer for efficient data processing and integration. Strong coding skills in Python (Pandas) and PySpark: Essential experience in using Python (Pandas) and PySpark for data manipulation, transformation, and pipeline development within Palantir Foundry, ensuring high performance and scalability ...

focus on designing scalable data platforms and integrating Generative AI capabilities into enterprise systems. Key Responsibilities Design, build and maintain scalable data pipelines using PySpark, Python and distributed computing frameworks Architect and optimise AWS-based data and AI infrastructure for secure, high-performance data processing Develop, fine-tune, benchmark … functional teams to integrate GenAI solutions into production environments Ensure data quality, governance and operational reliability across systems Essential Skills & Experience Strong experience with PySpark and large-scale distributed data processing (ETL/ELT pipelines) Advanced SQL expertise, including schema design (star/snowflake), indexing and optimisation techniques Proven ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

Role Overview The role is responsible for designing, building & maintaining robust data architectures that integrate SAP and non-SAP systems to support enterprise-wide data management, analytics, and reporting needs. This role ensures seamless data ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

native architectures, with experience in the financial sector. Experience with CI/CD , microservices , and event-driven architectures is essential, and knowledge of Python (pyspark) for data-driven workflows is advantageous Key Responsibilities Design, develop, and maintain full stack applications using React JS , C#.NET Core , and supporting technologies. Build … C#, REST APIs, Web APIs, and microservices-based development. Experience in transforming monolithic applications into composable microservices, or modular monoliths is a plus. Python (pyspark): Some hands-on experience with distributed data processing, data pipelines, and analytic transformations. Power BI: Some experience developing dashboards, data models, DAX calculations ...

Novatus Global is a Series B scale-up RegTech SaaS provider and boutique advisory firm, helping financial institutions manage their most complex regulatory requirements. We combine deep consulting expertise with cutting-edge SaaS solutions, enabling ...

solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards. Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services. Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing … Data Engineering, or a related field. Proven experience designing and building cloud-based data platforms, ideally within Azure. Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory. Solid understanding of Data Lakehouse architecture and modern data platform design. Proficiency in Python for data engineering, automation ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third‐party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd’s Syndicates Delegated ...

environments. Key Responsibilities Design and build scalable cloud-based data platforms using Medallion Architecture (Bronze/Silver/Gold) Engineer pipelines using Python and PySpark on platforms such as Databricks Ingest and integrate data from multiple sources including policy, claims, broker, and third-party systems Contribute to CI/… business stakeholders Required Experience Strong experience as a Senior Data Engineer within Insurance, ideally the London Market Proven hands-on expertise in: Python & PySpark Databricks Cloud platforms (Azure, AWS, or GCP) Solid understanding of Medallion Architecture and analytics-focused data modelling Strong domain knowledge across: Lloyd's Syndicates Delegated ...

Data Engineer l Azure l Fabric l Pyspark l Greenfield l 6-Month Contract | £475-£500 per day | Outside IR35 l Hybrid 3 days in London We're partnering with a leading London-based organisation that's building a modern, enterprise-wide data platform on Microsoft Fabric … Fabric Solid background in data modelling and architecture (Star Schema, Data Vault, Medallion), with a focus on performance and governance Hands-on expertise with PySpark, Spark SQL, CI/CD, Git, and working closely with BI teams on Power BI semantic models Experience contributing to platform standards and strategy ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

Role GenAI Data Engineer Your responsibilities: Design and maintain scalable data pipelines using PySpark, Python, and distributed computing frameworks to support high‐volume data processing. Architect and optimize AWS-based data and AI infrastructure, ensuring secure, performant, and cost‐efficient ingestion, transformation, and storage. Develop, finetune, benchmark, and evaluate … capabilities into production systems and ensure high-quality data, governance, and operational reliability Your Profile Essential skills/knowledge/experience: Strong experience with PySpark, distributed data processing, and largescale ETL/ELT pipelines. Strong SQL expertise including star/snowflake schema design, indexing strategies, writing optimized queries ...