12 of 12 Spark SQL Jobs in London

operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance, resilience, and data quality. Support reporting and MI use cases, including … equivalent. Active SC clearance (BPSS minimum). Strong hands-on experience with Microsoft Fabric and Azure data services. Advanced skills in PySpark, Spark SQL, Python, and SQL . Experience delivering data engineering solutions in government or similarly regulated environments. CI/CD, DevOps ...

Developing and deploying the Lakehouse platform using Databricks Building automation for cluster management, integration runtimes, and infrastructure components Leading implementation with Databricks, PySpark, Spark SQL, and Azure Data Factory Creating APIs for integration and automation Driving CI/CD, automated testing, and DevOps best practice Your … Skills and Experience You will have strong capability in: Azure and Databricks engineering PySpark, SQL, Python, and Spark SQL DevOps best practices including CI/CD, automation, and testing Infrastructure as code such as Terraform Stakeholder communication and cross-team collaboration How to Apply ...

data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data … routes Required Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ...

combat travel-based issues and advice on expected interview topics/questions. Main Duties and Responsibilities Write clean and testable code using SQL and Python scripting languages, to enable our customer data products and business applications Build and manage data pipelines and notebooks, deploying code in a structured … Promote an innovative thinking process and encourage it in others. Working within the agile framework at Chambers Skills and Experience Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with ...

Experience 5+ years' experience working as a Data Engineer or in a similar role Proven experience with cloud-based data platforms (Azure, AWS, SQL, Snowflake, Springserv); Strong proficiency in Spark SQL and PySpark , including complex transformations Experience building ETL/ELT pipelines using tools ...

Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing CI/CD pipelines (Azure DevOps, branching/merging, automation). Solid knowledge of data governance, lineage, access ...

maintain cloud data assets using Azure Databricks, Synapse and Data Lake technologies. Deliver real-time data processing using Kafka, Event Hub and Spark Streaming. Establish and maintain data governance, quality frameworks and PHI/PII compliance. Collaborate with stakeholders to design and deploy a Power BI 'Master Dashboard … semantic modelling. Required Skills & Experience Extensive experience with Azure Databricks, Azure Data Factory, Synapse and Unity Catalog Strong proficiency in Python, PySpark and Spark SQL Demonstrated experience with high-frequency time-series data Expertise in Star Schema, Snowflake Schema and Lakehouse architectures Experience with Great Expectations ...

London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record of ownership, delivery and operating effectively in fast-paced environments. What ...

role, focused on building, optimising, and operating Databricks-based Lakehouse solutions across cloud environments. Responsibilities: Design, build, and maintain data pipelines using Databricks (Spark, Delta Lake). Develop batch and streaming workloads using PySpark/Spark SQL. Implement data ingestion, transformation, and orchestration pipelines. Optimise Databricks … Databricks with cloud-native services across Azure, AWS, or GCP. Skills: Strong hands-on expertise with Databricks in production environments. Advanced PySpark and Spark SQL development experience. Experience building lakehouse architectures using Delta Lake. Strong cloud experience in Azure (preferred), AWS, or GCP. Experience with orchestration ...

experience in a data science or statistical role. Expertise in advanced analytics techniques including machine learning, optimisation and segmentation. Strong proficiency in Python, SQL and Spark SQL. Experience working with large datasets and distributed computing tools such as Azure Synapse and Snowflake. Familiarity with data transformation ...

Role You’ll design and deliver modern, cloud-based data solutions across large Insurance transformation programmes, working hands-on with: Azure Databricks & Spark Azure Data Factory, Synapse & Data Lake Large-scale ETL/ELT pipelines Enterprise data modelling & governance frameworks Technical leadership and delivery assurance What … experience (Policy, Claims, Regulatory Data) ✔ 8–10+ years in Data Engineering/Integration ✔ Deep expertise in Azure & Databricks ✔ Strong Spark (PySpark/SparkSQL) skills ✔ Experience designing scalable, high-performance data platforms ✔ Confident with SQL, CI/CD & DevOps practices Why Join? You’ll be part ...

SAS2PY) and manual refactoring. Pipeline Engineering: Design, build, and troubleshoot complex ETL/ELT workflows and data marts on AWS. Performance Tuning: Optimise Spark workloads for execution efficiency, partitioning, and cost-effectiveness. Quality Assurance: Implement clean coding principles, modular design, and robust unit/comparative testing to ensure …/CD integration, and comprehensive technical documentation. Technical Requirements PySpark (P3): 5+ years of hands-on experience writing scalable, production-grade PySpark/Spark SQL. AWS Data Stack (P3): Strong proficiency in EMR, Glue, S3, Athena, and Glue Workflows. SAS Knowledge (P1): Solid foundation in SAS to enable ...