20 of 20 Spark SQL Jobs in London

Microsoft Fabric Data Engineer

Location
Greater London, England, United Kingdom
industries.ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … error handlingSet up ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules ...

Data Lineage & Governance Analyst

Location
Greater London, England, United Kingdom
Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

Senior Data Engineer

Hiring Organisation
Hireful
Location
Central London, South East, United Kingdom
Employment Type
Permanent Contract, On-site
Salary
£89,999
extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

Senior Data Engineer - Databricks

Hiring Organisation
hireful
Location
London, United Kingdom
Employment Type
Permanent
Salary
£80000 - £90000/annum £80K - £90K Basic + 10% + Extensive
extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

Data Architect (Azure Databricks, Specialty Insurance)

Location
Greater London, England, United Kingdom
claims, exposure, reinsurance and finance. Design gold‐layer (consumption) models and, where Power BI is the consumption layer, semantic models served from Databricks SQL and shaped for efficient reporting and self‐service analytics. Select and design for the right storage and exchange formats for each workload, including … Parquet, CSV, Excel, XML, JSON, Protobuf). Optimise platform performance and cost, including partitioning vs. liquid clustering, file sizing, Photon, and cluster and SQL warehouse sizing. Manage the model lifecycle: optimise and update models for new and existing client projects, reverse engineer physical models from existing databases ...

Senior Data Engineer, Fabric & Analytics (Remote)

Location
Greater London, England, United Kingdom
data-driven culturewithin the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice You Will Demonstrate: Proficiencyin SQL and Python:You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data Lakehouse … Architecture:You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions:Youpossessexperience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or evenon-premiseSpark clusters, usingPySparkor Spark SQL to manage ...

Azure Data Engineer (Fabric & Cloud Data Platform)

Location
Greater London, England, United Kingdom
conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

Databricks Data Engineer

Hiring Organisation
GIS Recruiter Ltd
Location
London, United Kingdom
Employment Type
Permanent
Salary
£75000 - £95000/annum Hybrid Working
cloud-based data architectures. Essential Technical Skills 5+ years of hands-on experience with Databricks. Expert-level knowledge of Azure Databricks. Advanced PySpark, Spark SQL and Python programming. Strong experience developing complex ETL/ELT pipelines. Deep understanding of Delta … Lake, Delta Tables and Delta Live Tables/Lakeflow Declarative Pipelines. Experience implementing Medallion Architecture (Bronze, Silver and Gold). Strong knowledge of Apache Spark architecture, optimisation and performance tuning. Hands-on experience with Unity Catalog, access controls and data governance. Experience implementing incremental data loads ...

Microsoft Fabric Solution Architect

Location
Greater London, England, United Kingdom
capacity (F-SKU) sizing and licensingDesign end-to-end Microsoft Fabric solutions: OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL), Real-Time Intelligence (Eventstreams, Eventhouse/KQL) and Power BI + Power BI CoPilotDesign governance, security and operating frameworks: data governance … operating and support models and handover documentationPlan and run migrations from older platforms (Azure Synapse, Azure Data Factory, SSIS/SSAS, on-premises SQL Server, Snowflake, Databricks, Power BI Premium P-SKUs, Tableau/Qlik, etc.) to FabricPlan Fabric capacity (F-SKUs) and look after cost, performance ...

Data Engineering Manager

Location
City Of London, England, United Kingdom
leading multi‐disciplinary engineering teams. Solid understanding of the software and data delivery lifecycle. Technical Skills Strong experience with Databricks (PySpark/Spark SQL, Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows). Working knowledge of Microsoft Dynamics 365 data structures, Dataverse, and common integration patterns ...

Senior Data Analyst

Location
Greater London, England, United Kingdom
quantify their impact on migration. Design repeatable comparisons and reconciliation checks to validate that migrated data is complete, consistent and faithful. Write reusable SQL and Python scripts for profiling, exception detection and validation, using Azure Databricks, Git and dbt where relevant. Communicate findings clearly to technical … approach to working with data. Experience profiling unfamiliar datasets and tracing data quality issues to their root cause in a migration context. Strong SQL skills for complex joins, window functions, anomaly detection and source-to-target reconciliation. Working knowledge of Python, Azure Databricks or Spark SQL ...

QA Engineer

Location
Greater London, England, United Kingdom
named, specified, and tracked — not just worked around QA coverage expands naturally with each deployment increment rather than lagging behind Technical Strong SQL skills — comfortable writing and reading complex analytical queries (window functions, CTEs, aggregations) to interrogate data and verify correctness Hands-on experience with Databricks — running queries … navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

Data Engineer - Azure Databricks

Hiring Organisation
83zero Limited
Location
City of London, South East, United Kingdom
Employment Type
Permanent Contract, Hybrid
optimise ETL/ELT processes, integrating data from APIs, databases and other external sources. Work extensively with Databricks Unity Catalog, Delta Lake and Spark SQL. Develop complex data transformations, ensuring data accuracy, consistency and performance. Implement data quality checks, monitoring and validation processes. Utilise Microsoft Purview for data … Azure Databricks experience. Strong programming skills in Python, PySpark and SQL. Extensive experience with Azure Data Factory, Azure Blob Storage and Azure SQL Database. Strong understanding of Databricks Unity Catalog, Delta Lake and Spark SQL. Experience designing and optimising large-scale data pipelines. Strong knowledge ...

Databricks Solution Architect

Hiring Organisation
GIS Recruiter Ltd
Location
London, United Kingdom
Employment Type
Permanent
Salary
£90000 - £120000/annum Hybrid Working
Databricks platform experience. Extensive Azure Databricks architecture experience. Expert-level PySpark, Spark SQL and Python knowledge. Deep understanding of Apache Spark processing architecture. Extensive Delta Lake and Lakehouse architecture experience. Strong experience designing Medallion Architecture. Expert knowledge of Unity Catalog, governance, permissions and data ...

Principal Data Engineer

Location
Greater London, England, United Kingdom
with Pythonic data processing tools such as Pyspark and SparkSQL. Strong understanding of data modelling concepts relational and non-relational databases. Proficiency in SQL and experience with data integration and ETL tools. Excellent knowledge of big data technologies and frameworks. Strong knowledge of data governance and data security ...

Data Analyst

Hiring Organisation
Arcas Technology Ltd
Location
London, City of London, United Kingdom
Employment Type
Permanent
Salary
£45000 - £50000/annum
dashboards, including data models and DAX measures Working with stakeholders to understand how they use data and where processes can be improved Writing SQL to query, shape and analyse data from a range of sources Contributing to the Microsoft Fabric rollout (lakehouse, warehouse, OneLake) Presenting findings clearly … data analyst or similar role Strong Power BI skills, including DAX and data modelling (star schema, fact and dimension tables) Good SQL skills Advanced Excel (Power Query, pivots, lookups) A proactive, curious approach, with the confidence to engage with people across the business Good communication skills ...

Senior Applied AI Scientist

Location
Greater London, England, United Kingdom
translate technical content in commercial terms for non-technical stakeholders, such as customers, sales and product teams Desirable B2B product experience Experience with Spark SQL, PySpark, databricks Experience with CausalAI modelling frameworks (e.g. dowhy, econml) Diversity & Inclusion Intent HQ is an equal opportunities employer with ...

Principal Data Engineer

Location
London Borough of Lambeth, England, United Kingdom
Azure DataEngineering or Databricks preferred. Strong knowledge of Azuretechnologies, including ADF, ADLS, Databricks, Stream Analytics, DeltaLive Tables, and Synapse Analytics. Advanced proficiency in SQL andprogramming languages such as Python, Scala, or Spark-SQL. Experience in designing anddeveloping scalable ETL pipelines and optimizing data workflows. Deep understanding ...

Senior Data Scientist

Location
Greater London, England, United Kingdom
shape our core modeling practices. Your qualifications and experience ESSENTIAL Pro at using SQL for data manipulation (PostgreSQL, MSSQL, Google BigQuery, SparkSQL, etc.) Strong coding skills in Python, R, Scala, C, Java, or JS Proven track record of using data manipulation and machine learning libraries Practical modelling & statistical … apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

Data Analyst

Location
Greater London, England, United Kingdom
paced trading environment where insight is key to strategy Skills and experience you need as Data Analyst: Proficient in SQL & Python. PySpark & SparkSQL a bonus. Strong data viz skills using tools for presenting results such as Tableau or Power BI. Ability to write efficient code is essential ...