16 of 16 Permanent PySpark Jobs in London

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations ...

services such as Synapse, Data Factory, Databricks or Fabric Strong SQL skills with experience building optimised, complex queries Proficiency in Python and/or PySpark for data transformation and automation Strong experience with Power BI and modern data modelling techniques Demonstrable experience leading data engineering initiatives or projects Excellent ...

Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability to design, build, and maintain scalable data pipelines for batch and streaming data using modern ...

experience with cloud-based data platforms (Azure, AWS, SQL, Snowflake, Springserv); Microsoft Fabric experience is a strong plus Strong proficiency in Spark SQL and PySpark , including complex transformations Experience building ETL/ELT pipelines using tools such as Azure Data Factory or equivalent Ability to write efficient, reusable scripts ...

Designing and building cloud based data and machine learning pipelines that prepare data for analytics, AI, and product use ·Writing clear, well-structured Python, PySpark, and SQL to transform and validate data from multiple upstream sources ·Taking ownership of data quality, consistency, and reliability across the pipeline lifecycle ·Shaping ...

enquire about other roles, please contact Nick Mandella at Harnham. KEYWORDS Python, SQL, AWS, GCP, Azure, Cloud, Databricks, Docker, Kubernetes, CI/CD, Terraform, Pyspark, Spark, Kafka, machine learning, statistics, Data Science, Data Scientist, Big Data, Artificial Intelligence, private equity, finance. ...

DevOps and broader software engineering standards. Cloud experience (any platform). Previous mentoring experience. Nice to have: Snowflake or Databricks Spark, PySpark, Hadoop or similar big data tooling BI exposure (PowerBI, Tableau, etc.) Interview Process The process is fully structured, transparent, and efficient: Video call - high-level overview ...

DevOps and broader software engineering standards. Cloud experience (any platform). Previous mentoring experience. Nice to have: Snowflake or Databricks Spark, PySpark, Hadoop or similar big data tooling BI exposure (PowerBI, Tableau, etc.) Interview Process The process is fully structured, transparent, and efficient: Video call - high-level overview ...

Experience 5+ years in data engineering, including experience in a senior or lead position Strong expertise across AWS Hands-on experience with Databricks, PySpark, and Lakehouse patterns Strong Python and SQL engineering ability Experience implementing scalable ETL/ELT pipelines Background in data modelling and building analytics-ready datasets ...

timely manner. Key Experience, Skills and Knowledge: Proven 2 yrs+ experience as a Data Engineer or similar role, with a strong focus on PySpark, SQL, Microsoft Azure Data platforms and Power BI an advantage Proficiency in development languages suitable for intermediate-level data engineers, such as: Python/PySpark ...

timely manner. Key Experience, Skills and Knowledge: Proven 2 yrs+ experience as a Data Engineer or similar role, with a strong focus on PySpark, SQL, Microsoft Azure Data platforms and Power BI an advantage Proficiency in development languages suitable for intermediate-level data engineers, such as: Python/PySpark ...

Role and Responsibility Design, build and maintain scalable data pipelines and products within a modern Databricks Lakehouse environment using Delta Lake, Spark, Python (PySpark) and SQL. Manage and optimise the cloud-based lakehouse, leveraging Unity Catalogue, Databricks Workflows and best practices for performance, quality, lineage and compliance. Work closely ...

engineering is done across the organisation. Skills & Experience Strong hands-on experience with Azure data platforms Advanced SQL skills Commercial experience using Databricks and PySpark Proven background building and maintaining scalable data pipelines If you're looking for a role where you can combine technical depth, ownership, and influence ...

tagging, monitoring and platform operations. * Architect and design end-to-end data platform solutions focused on scalability, security and reliability. * Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. * Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. * Collaborate closely with data … followed. Your Skills and Experience You will bring: * Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. * Proficiency in PySpark, Python, SQL and Spark SQL. * Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. * Experience with monitoring, logging and alerting solutions. ...

Lakehouse solutions across cloud environments. Responsibilities: Design, build, and maintain data pipelines using Databricks (Spark, Delta Lake). Develop batch and streaming workloads using PySpark/Spark SQL. Implement data ingestion, transformation, and orchestration pipelines. Optimise Databricks workloads for performance, reliability, and cost. Integrate Databricks with cloud-native services … across Azure, AWS, or GCP. Skills: Strong hands-on expertise with Databricks in production environments. Advanced PySpark and Spark SQL development experience. Experience building lakehouse architectures using Delta Lake. Strong cloud experience in Azure (preferred), AWS, or GCP. Experience with orchestration tools (ADF, Airflow, dbt, or similar). Certifications ...

champion best practice methods; upskill the team through coaching and code reviews. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Communicate pricing proposals and insights clearly and persuasively to stakeholders at all levels. Collaborate with Modelling, Optimisation … insurance pricing. Strong technical expertise in predictive modelling techniques (e.g. GLMs and machine learning models). Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software and optimisation tools. A quantitative degree ...