1 to 25 of 61 PySpark Jobs in London

Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

needed: • Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory.• Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing.• Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability to design, build, and maintain scalable data pipelines for batch and streaming data using modern ...

Engineer or in a similar role Proven experience with cloud-based data platforms (Azure, AWS, SQL, Snowflake, Springserv); Strong proficiency in Spark SQL and PySpark , including complex transformations Experience building ETL/ELT pipelines using tools such as Azure Data Factory or equivalent Ability to write efficient, reusable scripts ...

culture of learning, quality, and continuous improvement. Essential Skills Strong experience with Azure Databricks and cloud data platforms. Advanced proficiency in Python, PySpark, and SQL. Experience developing Spark/Databricks pipelines. Hands-on experience with structured streaming and event-driven systems. Strong understanding of Lakehouse architecture and best practices. ...

Designing and building cloud based data and machine learning pipelines that prepare data for analytics, AI, and product use ·Writing clear, well-structured Python, PySpark, and SQL to transform and validate data from multiple upstream sources ·Taking ownership of data quality, consistency, and reliability across the pipeline lifecycle ·Shaping ...

across the platform. Key Skills & Experience Strong experience with Azure Databricks, Azure Data Factory and Azure Data Lake . Advanced Python, SQL and Spark (PySpark) development experience. Experience building and optimising ETL/data pipelines in cloud environments. Knowledge of CI/CD and version control (Azure DevOps, GitHub ...

experience required Strong experience in data engineering Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. ...

experience required Strong experience in data engineering Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. ...

Lloyd's of London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record of ownership, delivery and operating effectively in fast-paced environments. What ...

Maintain documentation, lineage, governance standards. Required Skills/Experience The ideal candidate will have the following: Strong SQL; Snowflake ELT; dbt experience. Python/PySpark, ETL/ELT design. Data Vault 2.0 or dimensional modeling. AWS services (S3, Glue, Lambda, Redshift) or GCP equivalents. Experience with CI/ ...

CDF. High technical proficiency with Cosmos DB SDKs (e.g., .NET, Java, Python, Node.js) Thorough understanding of Spark distributed computing concepts. High technical proficiency in PySpark and Spark Concepts. Experience with concurrency patterns, CLR, and scalable application design. Deep understanding of Azure services including Azure Functions, App Services ...

data pipelines • Engineering ML systems on modern cloud platforms • Working with architects and data engineers on enterprise AI platforms Technology • Python • SQL • Spark/PySpark • TensorFlow, PyTorch or similar frameworks • AWS, Azure or GCP • Databricks or Snowflake • Exposure to GenAI or LLMs Background We’re interested in engineers with ...

Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance, resilience, and data quality. Support reporting and MI use cases, including transformations and data … Data Engineer or equivalent. Active SC clearance (BPSS minimum). Strong hands-on experience with Microsoft Fabric and Azure data services. Advanced skills in PySpark, Spark SQL, Python, and SQL . Experience delivering data engineering solutions in government or similarly regulated environments. CI/CD, DevOps, and IaC experience ...

Lead Data Engineer - Pyspark/AWS/Python/SAS - Financial Sector As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale … PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment. Essential Skills PySpark & Data Engineering Minimum 5+ years of hands-on PySpark experience. SAS to Pyspark migration experience Proven ability ...

Hands-on experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory • Hands-on data engineering experience (Spark/PySpark, Delta Lake) • Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC • Experience leading Agile teams and managing risks, dependencies ...

Hands-on experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory• Hands-on data engineering experience (Spark/PySpark, Delta Lake)• Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC• Experience leading Agile teams and managing risks, dependencies ...

improvement initiatives to uplift practices and delivery maturity. Required Technical Skills & Expertise AWS Data & Analytics (Hands-On Expertise) AWS Glue (ETL/ELT development, PySpark, Glue Catalog) Apache Spark AWS Redshift (modelling, optimisation) AWS Lambda AWS RDS AWS Step Functions Experience with Open Table Formats: Delta Lake Apache Hudi ...

TensorFlow Serving, TorchServe). Familiarity with ML lifecycle tools such as MLflow, Kubeflow, or Azure ML Pipelines. Experience working with distributed data processing (e.g., PySpark) and SQL. Understanding of software engineering best practices, including version control (Git). Knowledge of CI/CD principles in ML environments. Experience with ...

basis, clarifying ownership and integration points, and capturing this information in a way that supports risk assessment and decommissioning decisions. Responsibilities: Python and PySpark are required as supporting capabilities, used where needed to analyse data pipelines and confirm how data moves and transforms in practice. The role also requires ...

Matplotlib) Data visualisation using Power BI or Tableau ideal Familiarity with Agile delivery methodologies Microsoft Fabric Power Apps, Power Automate, Dataverse Apache Spark/PySpark ...

design and optimisation within Microsoft Fabric Lakehouse, including Data Pipelines, Notebooks, and cloud-based ingestion/transformation layers. Deliver complex SQL, Python, and PySpark solutions for large-scale distributed processing. Coordinate workloads across BI developers and mentor junior and mid-level team members. Ensure BI quality, governance, documentation ...

scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical leadership … Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical leadership … Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

data modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...