25 of 25 Spark SQL Jobs in London

Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

knowledge‐sharing sessions to uplift team capability. Proven experience working as an Analytics Engineer, Data Engineer, or similar role. Deep proficiency in SQL for data transformation and analysis. Strong hands‐on experience with dbt (Data Build Tool) on Databricks, including advanced modelling techniques, test writing, and performance optimisation. … projects. Tools & Technologies You’ll Work With: Databricks (primary data platform for modelling and transformation) dbt for data modelling and transformation on Databricks SQL (Databricks SQL/Spark SQL) VS Code UI for SQL and dbt coding Git/GitHub ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

Security In-depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream ...

Security In-depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream ...

Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing ...

automation programs (preferably in financial services). Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet. Advanced SQL skills; experience with Oracle databases. Experience designing ...

structured governance. Strong domain understanding of Capital Markets data and reporting expectations (front-to-back awareness is a plus). Proficiency in SQL (advanced querying, performance tuning, reconciliation logic). Strong proficiency in Python for data analysis and automation (pandas, validation frameworks, scripting). Experience supporting or validating … outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Exposure to data governance concepts: critical data elements (CDEs), lineage, data quality dimensions, audit ...

plus). Familiarity with risk appetite concepts as applied to data quality thresholds and control exceptions. Technical/Analytical Skills Proficiency in SQL (advanced querying, reconciliation logic, data validation). Strong proficiency in Python for data analysis and automation (pandas, validation frameworks, scripting). Experience supporting or validating … outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Hands-on exposure to DCRM tooling and operational exception management processes. Experience with governance ...

relevant in data governance, compliance, or automation programs. Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet . Advanced SQL skills; experience with Oracle databases ...

named, specified, and tracked — not just worked around QA coverage expands naturally with each deployment increment rather than lagging behind Technical Strong SQL skills — comfortable writing and reading complex analytical queries (window functions, CTEs, aggregations) to interrogate data and verify correctness Hands-on experience with Databricks — running queries … navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

current as new views and data sources are onboarded; Engage with AI agents for test authoring, investigation, result analysis, and documentation. Требования Strong SQL skills, including window functions, CTEs, and aggregations; Hands-on experience with Databricks, Unity Catalog, and Spark job outputs; Working knowledge of PySpark … Spark SQL; Understanding of Lakehouse and medallion architecture; Familiarity with YAML-based configuration and structured test definitions; Comfortable with Git and basic engineering practices; Experience with AI-assisted workflows and large language model agents; 3-5+ Years of experience in data quality, data testing, analytics ...

your duties, you will be responsible for: Design, build, and deploy scalable data pipelines using Azure Databricks , Azure Data Factory , and Azure SQL Database . Optimise Databricks environments for performance, cost, and security. Develop and maintain data integration , ETL/ELT processes, and data quality frameworks. Translate business … environments. 8–10 years’ experience in data engineering and integration , ideally in enterprise‐scale environments. Hands‐on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working ...

your duties, you will be responsible for: Design, build and deploy scalable data pipelines using Azure Databricks , Azure Data Factory and Azure SQL Database . Optimise Databricks environments for performance, cost and security. Develop and maintain data integration , ETL/ELT processes and data quality frameworks. Translate retail … environments. 8-10 years' experience in data engineering and integration, ideally in enterprise-scale environments. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working with ...

/CD and platform engineering standards using Databricks Asset Bundles (DABs), Terraform and Azure DevOps or GitHub Actions Lead migrations from legacy platforms (SQL Server, Oracle, SAS, Hadoop, Synapse) with clear cut-over and reconciliation plans Monitor and optimise DBU consumption and cluster policies, giving the client clear … lead or architect Deep Unity Catalog experience: you have configured it, not just used it, including RBAC on sensitive data Strong PySpark, Spark SQL and Delta Lake, including performance tuning at scale Production pipeline experience with Lakeflow/DLT, Workflows and DABs Desirable: Databricks Certified Data ...

emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building ...

drive to make a positive impact on the world Experience working within a finance team and senior stakeholders Proficiency in writing robust, performant SQL queries Some experience building robust data pipelines, using Python and dbt Strong analytical skills, commercial interest, and the ability to distinguish between what matters … tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here Wondering what the salary for this role is Just ask us! On a call with ...

experience in data/network security would be a nice-to-have Datadog/Grafana/Prometheus Data related products (Airflow, Jupyter, Spark, dbt etc) The projects will be varied and we're looking for someone who can work autonomously and proactively to scope problems and solve … formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data modelling SparkSQL for analytics Streamlit for data applications Are you ready for a career with us? We want to ensure you have the right tools and environment ...