5 of 5 Permanent Delta Lake Jobs in the South West

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

data concepts, including storage, processing, and modelling. Core requirements include:Databricks - hands-on experience building and managing workloads on the Databricks platform, including Delta Lake, Unity Catalog, and Declarative Automation BundlesPySpark - proficiency in writing and optimising PySpark code for data transformation and processingAzure - solid experience working within … Azure ecosystem, including services such as Azure Data Factory, Azure Data Lake Storage, and Azure DevOpsAzure SQL Server - experience with Azure SQL Server is important, as it forms the foundation of our existing data warehouse. You will be working across both our SQL Server environment and our Databricks platform ...

cloud databases Proven track record with complex data migration projects (terabyte+ datasets, multiple legacy source systems, structures and unstructured data) Proficiency with Parquet/Delta Lake or other modern data storage formats Experience with streaming architectures using Kafka, Event Hubs, or Kinesis for real-time data processing Knowledge ...

data platform and services using the below technologies: Python, pandas and SQL in Jupyter notebooks for analysis dbt and Databricks data platform with Delta Lake Lightdash as our BI tool and semantic layer Streamlit for interactive data applications Airflow for orchestration Kubernetes for application deployment CI/ ...

pipelines and data products using strong SQL skills, modern data engineering patterns, cloud-native services, Spark and Databricks capabilities including notebooks, jobs, workflows and Delta Lake.Work confidently with common data formats and storage patterns such as Parquet, Delta, CSV, JSON and structured/semi-structured datasets, ensuring data ...