1 to 25 of 26 Spark SQL Jobs in the UK

Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze ? Silver ? Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee, legacy operational files … SharePoint, SFTP, SQL, REST and GraphQL APIs using Azure Data Factory and metadata-driven patterns. Apply data quality and validation rules using Lakeflow Declarative Pipelines expectations. Curated Layers & Data Modelling Develop clean and conforming Silver & Gold layers aligned to enterprise subject areas. Contribute to dimensional modelling (star schemas ...

Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze Silver Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee, legacy operational files … SharePoint, SFTP, SQL, REST and GraphQL APIs using Azure Data Factory and metadata-driven patterns. Apply data quality and validation rules using Lakeflow Declarative Pipelines expectations. Curated Layers & Data Modelling Develop clean and conforming Silver & Gold layers aligned to enterprise subject areas. Contribute to dimensional modelling (star schemas ...

data architectures using Medallion (Bronze/Silver/Gold) patterns Build metadata-driven ingestion pipelines and transformation frameworks Develop advanced PySpark/Spark SQL notebooks for data cleansing and modelling Create production-ready semantic models and support BI teams Implement governance, security, and CI/… business requirements into technical solutions Essential Skills Strong experience with Microsoft Fabric workloads (Lakehouse, Data Factory, Pipelines, Notebooks, Semantic Models) Advanced PySpark/Spark SQL for large-scale transformations Integration with Dynamics 365, Dataverse, and/or Business Central CI/CD implementation using Azure DevOps ...

Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical decisions. Implementing … Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps/ ...

data architecture aligned with modern best practices. Building and managing ingestion pipelines using Databricks and related tools. Developing PySpark/Spark SQL notebooks for complex transformations and cleansing. Applying governance, security, and CI/CD best practices across cloud environments. Leading technical discussions and producing professional … documentation. To be successful in this role, you will have: Hands-on experience with Databricks including Unity Catalog. Strong PySpark/Spark SQL skills for large-scale transformations. Experience integrating with diverse data sources such as APIs, cloud storage and databases. Experience with the Azure cloud ...

modeling, ETL/ELT development, and collaborative engineering practices. Key Responsibilities * Design, develop, and maintain scalable data pipelines using Azure Databricks (Python, PySpark, SQL). * Build and optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including … Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with data architects, analysts, and business stakeholders to understand data requirements. * Optimize Databricks clusters, jobs, and runtimes for performance and cost ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice Requirements You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

optimize end-to-end data pipelines using Azure services Build and maintain scalable ETL/ELT workflows using Azure Databricks (PySpark/SQL) Develop and manage data warehouses and analytics solutions using Azure Synapse Analytics Ingest data from multiple sources (APIs, databases, files, streaming sources) into Azure data … operational processes Required Qualifications 3+ years of experience as a Data Engineer or in a similar role Strong experience with Azure Databricks (PySpark, Spark SQL) Hands-on experience with Azure Synapse Analytics (dedicated and/or serverless pools) Solid understanding of data warehousing concepts and dimensional ...

optimize end-to-end data pipelines using Azure services Build and maintain scalable ETL/ELT workflows using Azure Databricks (PySpark/SQL) Develop and manage data warehouses and analytics solutions using Azure Synapse Analytics Ingest data from multiple sources (APIs, databases, files, streaming sources) into Azure data … operational processes Required Qualifications 3+ years of experience as a Data Engineer or in a similar role Strong experience with Azure Databricks (PySpark, Spark SQL) Hands-on experience with Azure Synapse Analytics (dedicated and/or serverless pools) Solid understanding of data warehousing concepts and dimensional ...

quality data solutions. Key Responsibilities: Develop, optimise, and maintain ETL/ELT pipelines within Databricks Build reliable data ingestion frameworks using PySpark and Spark SQL Design well-structured data models across medallion/lakehouse architecture Work with DevOps teams to automate deployments using CI/ ...

plans and lead the development of data processing workflows. From a technical standpoint you must possess: Extensive Databricks experience including Unity Catolog and Spark SQL Strong programming skills, preferably in Python and SQL Strong knowledge and experience in Azure, including working with Azure Data ...

house team, this role could be the next step for you. Data Engineer Responsibilities: Design, build, and optimise data pipelines in Python, PySpark, SparkSQL, and Databricks. Ingest, transform, and enrich structured, semi structured, and unstructured data. Operate and support production grade data systems with strong observability and monitoring. Enable real ...

including pension scheme and PMI Role and Responsibilities Design, build, and deploy scalable data pipelines using Azure Databricks, Azure Data Factory, and Azure SQL Database Optimise Databricks environments for performance, cost efficiency, and security Develop and maintain ETL/ELT processes and data quality frameworks Translate business requirements … environments Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse, and Azure Data Lake Hands-on experience with Spark (PySpark/SparkSQL) and distributed data processing Strong data modelling skills (Dimensional, ODS, Data Vault) and data warehousing concepts Proficiency in SQL, version control ...