3 of 3 Contract Apache Spark Jobs in the North of England

Python and SQL; APIs; test automation; data structures; distributed processing; Git; CI/CD; containers; infrastructure awareness; production debugging and performance optimization. Data engineering: Spark or Databricks, Kafka or equivalent streaming, ETL/ELT, batch and real-time pipelines, data contracts, lineage, cataloguing, quality rules, and scalable cloud storage. ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

role combines L2/L3 production support and Data Engineering tasks. Databricks experience covering notebooks, scheduled jobs, cluster configuration, driver/executor logs, Spark UI + performance troubleshooting. Supporting + maintaining production data pipelines, including incident investigation, safe recovery, root cause analysis + permanent remediation. Experience of Spark ...