4 of 4 PySpark Jobs in Dorset

schema design Develop API connectors and ingestion pipelines using Python Implement integration services feeding into the Databricks environment Build data transformation logic using PySpark/Python (data cleansing, standardisation, validation) Configure monitoring, logging, and alerting Execute unit and integration testing across pipelines Core Skills: Azure Databricks Python Desirable ...

implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency. Build and maintain curated data models (bronze/silver/gold), data quality checks … Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM ...

implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency. Build and maintain curated data models (bronze/silver/gold), data quality checks … Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM ...

teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products. The role emphasizes batch/streaming pipelines, Spark/PySpark, Delta Lake, governance and security, with CI/CD and collaboration across environments. #J-18808-Ljbffr ...