1 to 25 of 35 Permanent Spark SQL Jobs in London

Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

technical SME across Databricks, Fabric, Synapse, and Azure. Own solution design, implementation strategy, and technical delivery. Build and optimise pipelines using PySpark , Spark SQL , ADF , Synapse Pipelines , or Fabric Pipelines . Define engineering best practice, conduct code reviews, and uplift quality. Develop secure, performant SQL … bring Proven leadership of Data Engineering teams. Strong commercial experience with Databricks , Microsoft Fabric , or Azure Synapse . Deep expertise in PySpark , Spark SQL , and SQL optimisation. ETL/ELT pipeline development and orchestration. Experience building scalable cloud-based data platforms. Nice to Have ...

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

bonus and an extensive benefits package. What you'll be doingDesign, build and maintain scalable data pipelines and data products using Databricks, PySpark, Spark SQL and Delta Lake.Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems.Shape enterprise … help mature a growing data function.What you need to bring“Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc)Advanced SQL and Python coding skills.A track record of building and optimising enterprise-scale data pipelines ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

transformation of its data estate. As part of a wider modernisation programme, the business is migrating critical real estate data capabilities from legacy SQL Server environments into a modern Databricks platform, creating a more scalable, governed and future-ready foundation for analytics, reporting and investment decision-making. This … platform architecture, data governance and reporting capabilities across a complex investment environment. What You'll Do Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment ...

technical stakeholders, including customers, sales, and product teams Experience with deep learning statistical models such as Transformers B2B product experience desirable Experience with Spark SQL, PySpark, and Databricks desirable Experience with CausalAI modelling frameworks such as DoWhy and EconML desirable Core Competencies Demonstrates expertise in designing … Science Techniques Behavioural Insights Soft Skills Creative Problem Solving Collaboration Communication Industry Keywords B2B Product Experience Agentic AI Behavioural Data Science Tools & Technologies Spark SQL PySpark Databricks #J-18808-Ljbffr ...

techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL.Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologiesProficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability.Designing and building solutions on a cloud data warehouse or lakehouse ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL. Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologies Proficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability. Designing and building solutions on a cloud data warehouse ...

cleansed, conformed business layers) while securely handling identifiable data.Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark.Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across …/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats.Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

Security In-depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream ...

Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing ...

Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into … need from you 8+ years in data engineering, including 3+ years hands on with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data quality and governance experience ...

structured governance. Strong domain understanding of Capital Markets data and reporting expectations (front-to-back awareness is a plus). Proficiency in SQL (advanced querying, performance tuning, reconciliation logic). Strong proficiency in Python for data analysis and automation (pandas, validation frameworks, scripting). Experience supporting or validating … outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Exposure to data governance concepts: critical data elements (CDEs), lineage, data quality dimensions, audit ...

automation programs (preferably in financial services). Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet. Advanced SQL skills; experience with Oracle databases. Experience designing ...

plus). Familiarity with risk appetite concepts as applied to data quality thresholds and control exceptions. Technical/Analytical Skills Proficiency in SQL (advanced querying, reconciliation logic, data validation). Strong proficiency in Python for data analysis and automation (pandas, validation frameworks, scripting). Experience supporting or validating … outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Hands-on exposure to DCRM tooling and operational exception management processes. Experience with governance ...

relevant in data governance, compliance, or automation programs. Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet . Advanced SQL skills; experience with Oracle databases ...

interested in engineers with strong hands-on Databricks experience. Essential Skills Commercial experience building solutions within Databricks. Strong hands-on expertise with PySpark, Spark SQL, and Delta Lake. Experience designing and supporting production-grade ETL/ELT pipelines. Strong Python and SQL development skills. ...

experience designing canonical data models or enterprise logical data models. Strong understanding of data modelling techniques, including conceptual, logical and physical modelling. Strong SQL skills and ability to interrogate complex relational data structures. Experience working with cloud data platforms, ideally AWS and Databricks. Understanding of modern data architecture … platforms. Experience with regulatory, client or operational reporting. Experience supporting data migration, legacy system modernisation or platform replacement. Experience with Databricks, Delta Lake, Spark SQL or PySpark. Experience defining validation frameworks, data quality dashboards or metadata-driven rule engines. Experience working in consultancy or client-facing ...

your duties, you will be responsible for: Design, build, and deploy scalable data pipelines using Azure Databricks , Azure Data Factory , and Azure SQL Database . Optimise Databricks environments for performance, cost, and security. Develop and maintain data integration , ETL/ELT processes, and data quality frameworks. Translate business … environments. 8–10 years’ experience in data engineering and integration , ideally in enterprise‐scale environments. Hands‐on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working ...

current as new views and data sources are onboarded; Engage with AI agents for test authoring, investigation, result analysis, and documentation. Требования Strong SQL skills, including window functions, CTEs, and aggregations; Hands-on experience with Databricks, Unity Catalog, and Spark job outputs; Working knowledge of PySpark … Spark SQL; Understanding of Lakehouse and medallion architecture; Familiarity with YAML-based configuration and structured test definitions; Comfortable with Git and basic engineering practices; Experience with AI-assisted workflows and large language model agents; 3-5+ Years of experience in data quality, data testing, analytics ...