13 of 13 Spark SQL Jobs in the UK

hands-on, focused on building and optimising large-scale data pipelines, dataflows, semantic models, and lakehouse components. Key Responsibilities Design, build and optimise Spark-based data pipelines for batch and streaming workloads Develop Fabric dataflows, pipelines, and semantic models Implement complex transformations, joins, aggregations and performance tuning Build … translate requirements into scalable technical solutions Troubleshoot and improve reliability, latency and workload performance Essential Skills Strong hands-on experience with PySpark, Spark SQL, Spark Streaming, DataFrames Microsoft Fabric (Fabric Spark jobs, dataflows, pipelines, semantic models) Azure: ADLS, cloud data engineering, notebooks ...

Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze ? Silver ? Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee, legacy operational files … SharePoint, SFTP, SQL, REST and GraphQL APIs using Azure Data Factory and metadata-driven patterns. Apply data quality and validation rules using Lakeflow Declarative Pipelines expectations. Curated Layers & Data Modelling Develop clean and conforming Silver & Gold layers aligned to enterprise subject areas. Contribute to dimensional modelling (star schemas ...

Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze Silver Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee, legacy operational files … SharePoint, SFTP, SQL, REST and GraphQL APIs using Azure Data Factory and metadata-driven patterns. Apply data quality and validation rules using Lakeflow Declarative Pipelines expectations. Curated Layers & Data Modelling Develop clean and conforming Silver & Gold layers aligned to enterprise subject areas. Contribute to dimensional modelling (star schemas ...

Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical decisions. Implementing … Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps/ ...

data architectures using Medallion (Bronze/Silver/Gold) patterns Build metadata-driven ingestion pipelines and transformation frameworks Develop advanced PySpark/Spark SQL notebooks for data cleansing and modelling Create production-ready semantic models and support BI teams Implement governance, security, and CI/… business requirements into technical solutions Essential Skills Strong experience with Microsoft Fabric workloads (Lakehouse, Data Factory, Pipelines, Notebooks, Semantic Models) Advanced PySpark/Spark SQL for large-scale transformations Integration with Dynamics 365, Dataverse, and/or Business Central CI/CD implementation using Azure DevOps ...

data architecture aligned with modern best practices. Building and managing ingestion pipelines using Databricks and related tools. Developing PySpark/Spark SQL notebooks for complex transformations and cleansing. Applying governance, security, and CI/CD best practices across cloud environments. Leading technical discussions and producing professional … documentation. To be successful in this role, you will have: Hands-on experience with Databricks including Unity Catalog. Strong PySpark/Spark SQL skills for large-scale transformations. Experience integrating with diverse data sources such as APIs, cloud storage and databases. Experience with the Azure cloud ...

Designing end-to-end data architecture aligned with modern best practices Building and managing ingestion pipelines using Fabric Data Factory Developing PySpark/Spark SQL notebooks for complex transformations and cleansing Applying governance, security, and CI/CD best practices across Fabric environments. Leading client workshops … successful in this role, you will have: Hands-on experience with Microsoft Fabric workloads (Lakehouse, Pipelines, Notebooks). Strong PySpark/Spark SQL skills for large-scale transformations. Experience integrating with a variety of different sources such as Dynamics 365, Dataverse or Business Central. Knowledge ...

modeling, ETL/ELT development, and collaborative engineering practices. Key Responsibilities * Design, develop, and maintain scalable data pipelines using Azure Databricks (Python, PySpark, SQL). * Build and optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including … Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with data architects, analysts, and business stakeholders to understand data requirements. * Optimize Databricks clusters, jobs, and runtimes for performance and cost ...

data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks. Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle. Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related services. Apply best practices for development, deployment, and workload optimization. Program in SQL, Python, R, YAML, and JavaScript. Integrate data from relational databases, APIs, and streaming sources using best-practice patterns. Collaborate with API developers for seamless data exchange. Utilize Azure ...

pipelines to ingest, transform, and load data from diverse sources (APIs, databases, files) into Azure Databricks. Implement data cleaning, validation, and enrichment using Spark (PySpark/Scala) and related tools to ensure quality and consistency. Utilize Unity Catalog, Delta Lake, Spark SQL, and best … practices for Databricks development, optimization, and deployment. Program in SQL, Python, R, YAML, and JavaScript. Integrate data from multiple sources and formats (CSV, JSON, Parquet, Delta) for downstream analytics, dashboards, and reporting. Apply Azure Purview for governance and quality checks. Monitor pipelines, resolve issues, and enhance data quality ...

plans and lead the development of data processing workflows. From a technical standpoint you must possess: Extensive Databricks experience including Unity Catolog and Spark SQL Strong programming skills, preferably in Python and SQL Strong knowledge and experience in Azure, including working with Azure Data ...

house team, this role could be the next step for you. Data Engineer Responsibilities: Design, build, and optimise data pipelines in Python, PySpark, SparkSQL, and Databricks. Ingest, transform, and enrich structured, semi structured, and unstructured data. Operate and support production grade data systems with strong observability and monitoring. Enable real ...

analytics; exposure to predictive modelling and machine learning is a plus. Proficient in SQL and Python, with bonus points for PySpark, SparkSQL, and Git. Skilled in data visualisation with tools such as Tableau or Power BI. Confident writing efficient code and troubleshooting sophisticated queries. Clear and adaptable communicator ...