5 of 5 Spark Streaming Jobs in the UK

gateway design, tenant isolation, and migration strategies for enterprise clients (parallel runs, phased cutovers). Modernize data architecture to support real-time streaming and graph-based modeling (Kafka, Flink) and unified analytics for operations and ML training. Transition release management from quarterly waterfall to continuous deployment with feature … multi-tenant isolation, Kubernetes/container orchestration, serverless patterns. Working knowledge of cloud data platforms (Snowflake, Databricks, BigQuery, Azure Synapse) and real-time streaming (Kafka, Flink, Spark Streaming). Familiarity with generative AI patterns (LLM fine-tuning, RAG, prompt engineering, function calling, agent frameworks ...

time insight into live positions and risk. This platform underpins critical trading decisions and represents a major step forward in scalability, resilience, and streaming architecture across our technology estate. Responsibilities Design and build distributed, real‐time streaming components within a Flink‐based architecture. Develop and maintain … detects and publishes meaningful changes in liability. Ensure correctness in a high‐throughput environment through robust handling of ordering, duplication, and idempotency. Optimise streaming jobs for latency, throughput, and resource efficiency. Collaborate closely with trading and product teams to translate domain requirements into scalable, reliable systems. Contribute ...

investing heavily in cloud technologies, scalable data infrastructure, and modern engineering practices. The Role and Deliverables Design, build, and operate high-volume streaming and batch data pipelines across a modern cloud environment. Deliver complex migration projects, including the consolidation of real-time analytics platforms onto a scalable architecture. … wider data team. Your Skills & Experience Strong experience owning and operating production data infrastructure at significant scale. Proven capability delivering large-scale streaming data solutions using technologies such as Kafka, Flink, Spark Streaming, Dataflow, or similar. Demonstrated success leading end-to-end migrations ...

Proven experience designing and building distributed systems and APIs at scale. Hands‐on experience with real‐time data processing frameworks (e.g. Kafka, Flink, Spark Streaming, Pulsar). Experience working with cloud platforms (AWS, GCP, or Azure) and containerisation/orchestration (Docker, Kubernetes). Strong understanding … data modelling, storage systems, and streaming/processing architectures. Familiarity with ML tooling such as MLflow, ZenML, or Metaflow. Familiarity with ML model lifecycle management (training, deployment, monitoring). Excellent collaboration and communication skills, with the ability to work cross‐functionally. Nice to have Experience building or working ...

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … practices. Architect and develop federated data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised ...