23 of 23 Spark SQL Jobs in London

industries.ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … error handlingSet up ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules ...

Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

claims, exposure, reinsurance and finance. Design gold‐layer (consumption) models and, where Power BI is the consumption layer, semantic models served from Databricks SQL and shaped for efficient reporting and self‐service analytics. Select and design for the right storage and exchange formats for each workload, including … Parquet, CSV, Excel, XML, JSON, Protobuf). Optimise platform performance and cost, including partitioning vs. liquid clustering, file sizing, Photon, and cluster and SQL warehouse sizing. Manage the model lifecycle: optimise and update models for new and existing client projects, reverse engineer physical models from existing databases ...

data-driven culturewithin the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice You Will Demonstrate: Proficiencyin SQL and Python:You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data Lakehouse … Architecture:You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions:Youpossessexperience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or evenon-premiseSpark clusters, usingPySparkor Spark SQL to manage ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

capacity (F-SKU) sizing and licensingDesign end-to-end Microsoft Fabric solutions: OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL), Real-Time Intelligence (Eventstreams, Eventhouse/KQL) and Power BI + Power BI CoPilotDesign governance, security and operating frameworks: data governance … operating and support models and handover documentationPlan and run migrations from older platforms (Azure Synapse, Azure Data Factory, SSIS/SSAS, on-premises SQL Server, Snowflake, Databricks, Power BI Premium P-SKUs, Tableau/Qlik, etc.) to FabricPlan Fabric capacity (F-SKUs) and look after cost, performance ...

leading multi-disciplinary engineering teams. Solid understanding of the software and data delivery lifecycle. Technical Skills Strong experience with Databricks (PySpark/Spark SQL, Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows). Working knowledge of Microsoft Dynamics 365 data structures, Dataverse, and common integration patterns ...

transformations, and quantify their impact on migration.Design repeatable comparisons and reconciliation checks to validate that migrated data is complete, consistent and faithful.Write reusable SQL and Python scripts for profiling, exception detection and validation, using Azure Databricks, Git and dbt where relevant.Communicate findings clearly to technical and non-technical … evidence-based approach to working with data.Experience profiling unfamiliar datasets and tracing data quality issues to their root cause in a migration context.Strong SQL skills for complex joins, window functions, anomaly detection and source-to-target reconciliation.Working knowledge of Python, Azure Databricks or Spark SQL ...

quantify their impact on migration. Design repeatable comparisons and reconciliation checks to validate that migrated data is complete, consistent and faithful. Write reusable SQL and Python scripts for profiling, exception detection and validation, using Azure Databricks, Git and dbt where relevant. Communicate findings clearly to technical … approach to working with data. Experience profiling unfamiliar datasets and tracing data quality issues to their root cause in a migration context. Strong SQL skills for complex joins, window functions, anomaly detection and source-to-target reconciliation. Working knowledge of Python, Azure Databricks or Spark SQL ...

named, specified, and tracked — not just worked around QA coverage expands naturally with each deployment increment rather than lagging behind Technical Strong SQL skills — comfortable writing and reading complex analytical queries (window functions, CTEs, aggregations) to interrogate data and verify correctness Hands-on experience with Databricks — running queries … navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

data quality and governance. Create and maintain ETL/ELT processes using Data Factory, Data Pipelines, Notebooks, and Spark. Develop, maintain, and enhance SQL scripts, stored procedures, views, and data models. Monitor and troubleshoot performance issues within Fabric environments. Implement security, access controls, and governance standards. Support … architects, and stakeholders to translate requirements into technical solutions. Produce technical documentation and knowledge transfer materials. What you will have Technical Skills Strong SQL development skills. Experience designing and implementing ETL/ELT solutions. Knowledge of Lakehouse and Data Warehouse architectures. Experience with Data Pipelines, Dataflows Gen2 ...

support data quality and governance.Create and maintain ETL/ELT processes using Data Factory, Data Pipelines, Notebooks, and Spark.Develop, maintain, and enhance SQL scripts, stored procedures, views, and data models.Monitor and troubleshoot performance issues within Fabric environments.Implement security, access controls, and governance standards.Support CI/CD deployment processes … business analysts, architects, and stakeholders to translate requirements into technical solutions.Produce technical documentation and knowledge transfer materials. What you will have Technical SkillsStrong SQL development skills. Experience designing and implementing ETL/ELT solutions. Knowledge of Lakehouse and Data Warehouse architectures. Experience with Data Pipelines, Dataflows Gen2 ...

critical to meeting Nomura’s regulatory and internal liquidity reporting requirements.Architect, develop, and enhance a large-scale data processing platform using Scala and Apache Spark on AWS \ MinIO.Work closely with business users to analyse business requirements and deliver high-quality data for regulatory reporting, including liquidity and risk … English with strong written & verbal communication skillsExcellent teamwork skills, comfortable with working as part of a globally spread-out teamPreferred:Scala ExperienceExperience with Apache Spark SQL/DataframesKnowledge of Basel III and liquidity reportingExposure and interest in using AI coding tools such as Claude Code ...

meeting Nomura’s regulatory and internal liquidity reporting requirements. Architect, develop, and enhance a large-scale data processing platform using Scala and Apache Spark on AWS \ MinIO . Work closely with business users to analyse business requirements and deliver high-quality data for regulatory reporting, including liquidity … written & verbal communication skills Excellent teamwork skills, comfortable with working as part of a globally spread-out team Preferred: Scala Experience Experience with Apache Spark SQL/Dataframes Knowledge of Basel III and liquidity reporting Exposure and interest in using AI coding tools such as Claude ...

meeting Nomura’s regulatory and internal liquidity reporting requirements. Architect, develop, and enhance a large-scale data processing platform using Scala and Apache Spark on AWS \ MinIO . Work closely with business users to analyse business requirements and deliver high-quality data for regulatory reporting, including liquidity … written & verbal communication skills Excellent teamwork skills, comfortable with working as part of a globally spread-out team Preferred: Scala Experience Experience with Apache Spark SQL/Dataframes Knowledge of Basel III and liquidity reporting Exposure and interest in using AI coding tools such as Claude ...

translate technical content in commercial terms for non-technical stakeholders, such as customers, sales and product teams Desirable B2B product experience Experience with Spark SQL, PySpark, databricks Experience with CausalAI modelling frameworks (e.g. dowhy, econml) Diversity & Inclusion Intent HQ is an equal opportunities employer with ...

Azure DataEngineering or Databricks preferred. Strong knowledge of Azuretechnologies, including ADF, ADLS, Databricks, Stream Analytics, DeltaLive Tables, and Synapse Analytics. Advanced proficiency in SQL andprogramming languages such as Python, Scala, or Spark-SQL. Experience in designing anddeveloping scalable ETL pipelines and optimizing data workflows. Deep understanding ...

shape our core modeling practices. Your qualifications and experience ESSENTIAL Pro at using SQL for data manipulation (PostgreSQL, MSSQL, Google BigQuery, SparkSQL, etc.) Strong coding skills in Python, R, Scala, C, Java, or JS Proven track record of using data manipulation and machine learning libraries Practical modelling & statistical … apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

paced trading environment where insight is key to strategy Skills and experience you need as Data Analyst: Proficient in SQL & Python. PySpark & SparkSQL a bonus. Strong data viz skills using tools for presenting results such as Tableau or Power BI. Ability to write efficient code is essential ...

drive to make a positive impact on the world Experience working within a finance team and senior stakeholders Proficiency in writing robust, performant SQL queries Some experience building robust data pipelines, using Python and dbt Strong analytical skills, commercial interest, and the ability to distinguish between what matters … tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here Wondering what the salary for this role is Just ask us! On a call with ...

experience in data/network security would be a nice-to-have Datadog/Grafana/Prometheus Data related products (Airflow, Jupyter, Spark, dbt etc) The projects will be varied and we're looking for someone who can work autonomously and proactively to scope problems and solve … formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data modelling SparkSQL for analytics Streamlit for data applications Are you ready for a career with us? We want to ensure you have the right tools and environment ...