1 to 25 of 31 Remote/Hybrid Spark SQL Jobs in the UK

with a target delivery date of the end of December. The successful consultant will be responsible for migrating an existing Azure Synapse Dedicated SQL Pool codebase into Databricks, ensuring the migrated solution delivers equivalent or improved performance, maintains data accuracy and supports ongoing business consumption. We are specifically … skills, including complex stored procedures, transformations and query optimisation. Deep understanding of MPP architecture, distributed tables, execution plans and performance tuning. Databricks & Apache Spark Extensive commercial experience with Azure Databricks, Spark SQL, Databricks SQL and PySpark. Strong understanding of Apache Spark ...

reporting, analytics, and operational decision-making across the project programme. Working within a multidisciplinary Data & Analytics team, the successful candidate will use Databricks, SQL, Python, and Microsoft technologies to deliver robust, scalable, and maintainable data solutions. They will collaborate closely with Data Architects and Performance Analysts to transform … business intelligence solutions. Strong proven commercial experience with Databricks. Strong experience developing ETL/ELT pipelines and data integration solutions. Experience using Apache Spark, PySpark, and Spark SQL. Strong SQL development and optimisation skills. Experience working with large and complex datasets from multiple source ...

scalable Data Engineering solutions. Experience of core MS Fabric features including: OneLake, Lakehouses, Warehouses, Data Pipelines, Dataflows, Gen2, Notebooks, Eventstreams + Eventhouses. Using SQL, T-SQL, Spark SQL, PySpark + Python for data ingestion, data transformation, data optimisation, data modelling, and converting ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

knowledge‐sharing sessions to uplift team capability. Proven experience working as an Analytics Engineer, Data Engineer, or similar role. Deep proficiency in SQL for data transformation and analysis. Strong hands‐on experience with dbt (Data Build Tool) on Databricks, including advanced modelling techniques, test writing, and performance optimisation. … projects. Tools & Technologies You’ll Work With: Databricks (primary data platform for modelling and transformation) dbt for data modelling and transformation on Databricks SQL (Databricks SQL/Spark SQL) VS Code UI for SQL and dbt coding Git/GitHub ...

techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory ...

3+ years with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema ...

3+ years with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema ...

lakehouses and data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured … semi-structured and unstructured data sources. Migration and modernisation of legacy platforms such as Oracle, SQL Server/SSIS or Synapse. Power BI semantic models, curated datasets and supporting analytics platforms. Data quality, governance, metadata management and lineage. Data security, classification and access control approaches. Automated testing, validation ...

platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured … semi-structured and unstructured data sources. Migration and modernisation of legacy platforms such as Oracle, SQL Server/SSIS or Synapse. Power BI semantic models, curated datasets and supporting analytics platforms. Data quality, governance, metadata management and lineage. Data security, classification and access control approaches. Automated testing, validation ...

platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured … semi-structured and unstructured data sources. Migration and modernisation of legacy platforms such as Oracle, SQL Server/SSIS or Synapse. Power BI semantic models, curated datasets and supporting analytics platforms. Data quality, governance, metadata management and lineage. Data security, classification and access control approaches. Automated testing, validation ...

team. This role is central to delivering Canopius' data strategy through development of new data capabilities and supporting the final transition from legacy SQL Server-based environments. Alongside hands-on delivery, the role provides technical leadership across data engineering initiatives, helping to establish a future-ready, cloud-native … secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines to ensure ...

processes and solutions. Lead the execution of Microsoft Fabric migration activities. Lead and guide migration engineers and QA resources. Develop Python, PySpark and Spark SQL code for data validation and curation within notebooks. Create data integration test cases and support testing activities. Implement and amend data … Azure Functions o Azure App Service o Logic Apps o Azure Kubernetes Service (AKS) o Web Apps Hands-on expertise in PySpark, SQL and Python. Good knowledge of real-time streaming applications, preferably including Kafka, Azure Functions or Azure Service Bus. Strong understanding of Spark processing ...

design and deliver cutting-edge data solutions for customers operating in some of the UK's most complex and challenging environments. Using Databricks, Apache Spark and cloud-native technologies, you’ll develop scalable Lakehouse architectures and production-grade data pipelines that enable organisations to trust, share and exploit … grow your career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable data products. ...

Databricks) Leeds (Hybrid) 6 Month Contract £(Apply online only)/day (Inside IR35) Data Engineer needed with strong hands-on Databricks, PySpark, AWS, SQL + Python experience. 6 Month Contract based in Leeds (Hybrid). Start ASAP in October 2026. Hybrid Working Model - primarily working from home with … work with a leading global IT transformation on a large-scale Government project: Data Engineer with strong Databricks, AWS, Python, PySpark + SQL skills. Hands-on Data Engineer to run + maintain production data pipelines on Databricks and AWS. Keeping scheduled data processing reliable, resolve incidents, fixes + ...

work with a leading global IT transformation on a large-scale Government project: Data Engineer with strong Databricks, AWS, Python, PySpark and SQL skills. Hands-on Data Engineer to run + maintain production data pipelines on Databricks and AWS. Keeping scheduled data processing reliable, resolve incidents, fixes + … role combines L2/L3 production support and Data Engineering tasks. Databricks experience covering notebooks, scheduled jobs, cluster configuration, driver/executor logs, Spark UI + performance troubleshooting. Supporting + maintaining production data pipelines, including incident investigation, safe recovery, root cause analysis + permanent remediation. Experience of Spark ...

named, specified, and tracked — not just worked around QA coverage expands naturally with each deployment increment rather than lagging behind Technical Strong SQL skills — comfortable writing and reading complex analytical queries (window functions, CTEs, aggregations) to interrogate data and verify correctness Hands-on experience with Databricks — running queries … navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

your duties, you will be responsible for: Design, build and deploy scalable data pipelines using Azure Databricks , Azure Data Factory and Azure SQL Database . Optimise Databricks environments for performance, cost and security. Develop and maintain data integration , ETL/ELT processes and data quality frameworks. Translate retail … environments. 8-10 years' experience in data engineering and integration, ideally in enterprise-scale environments. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working with ...

your duties, you will be responsible for: Design, build and deploy scalable data pipelines using Azure Databricks , Azure Data Factory and Azure SQL Database . Optimise Databricks environments for performance, cost and security. Develop and maintain data integration , ETL/ELT processes and data quality frameworks. Translate retail … environments. Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse and Azure Data Lake. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working with ...

/CD and platform engineering standards using Databricks Asset Bundles (DABs), Terraform and Azure DevOps or GitHub Actions Lead migrations from legacy platforms (SQL Server, Oracle, SAS, Hadoop, Synapse) with clear cut-over and reconciliation plans Monitor and optimise DBU consumption and cluster policies, giving the client clear … lead or architect Deep Unity Catalog experience: you have configured it, not just used it, including RBAC on sensitive data Strong PySpark, Spark SQL and Delta Lake, including performance tuning at scale Production pipeline experience with Lakeflow/DLT, Workflows and DABs Desirable: Databricks Certified Data ...

migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable ...

migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable ...

migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable ...

migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable ...

migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable ...