11 of 11 Permanent Batch Processing Jobs in England

strong fit. The Role Work with multi-terabyte to large-scale datasets (100TB+), primarily text-based Build and optimise data ingestion and processing pipelines Improve batch processing performance and throughput across large datasets Solve challenges related to scalability, reliability, and data flow Contribute to the development … optimise data pipelines at scale Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop Strong understanding of data ingestion and batch processing methodologies Experience with text or unstructured data is advantageous but not essential A hands-on, problem-solving mindset with a focus ...

self-service capabilities Your Responsibilities As a Senior Data Engineer , you will: Design and implement data ingestion pipelines using ETL, streaming, scraping, and batch approaches Clean, enrich, and transform datasets for analytical and operational consumption Persist data across databases, warehouses, and data lakes Distribute data internally via APIs, Python … packages, and direct querying Maintain and enhance production data pipelines and databases Support post-processing automation, including analytics, models, and visualisation workflows Enable Data Scientists through shared libraries, cloud resources, and documented data access Maintain knowledge bases covering data sources, pipelines, and usage Key Requirements As a Senior Data ...

sources and apis, with good understanding of event and message-based integrations using Active MQ/Kafka/AWS kinesis etc, as well as batch processes of synchronous integration involving change pointers and delta refresh · Designing reporting solutions with visualisation suite covering Tableau, PowerBI, SAP BOBJ, Bex, SAP analytics ...

expert IT users. Experienced in creating technical and process documentation Demonstrable practical experience of at least some of the following: Data Manipulation, Batch Processing, case flow management Power BI, Power Automate, SQL, Microsoft Fabric, ETL/ELT Pipelines Housing Management and/or large database systems management ...

enterprise databases into AWS using DMS and CDC patterns Ensure automation, scalability, and resilience across platform and data pipelines Enable event-driven and batch processing architectures (e.g., S3, Lambda) Enforce engineering standards and best practices across the platform Ensure data lineage, cataloguing, and governance are maintained Collaborate with ...

function, setting standards, mentoring engineers, and working closely with both internal teams and external delivery partners. The Role Data Engineering & Pipeline Development Transform legacy batch processes into scalable, observable data pipelines Build and deliver high-quality data products in line with architectural designs Work across modern cloud platforms including ...

function, setting standards, mentoring engineers, and working closely with both internal teams and external delivery partners. The Role Data Engineering & Pipeline Development Transform legacy batch processes into scalable, observable data pipelines Build and deliver high-quality data products in line with architectural designs Work across modern cloud platforms including ...

authority within the data function, setting standards, mentoring engineers, and working closely with both internal teams and external delivery partners. The Role Transform legacy batch processes into scalable, observable data pipelines Build and deliver high-quality data products in line with architectural designs Work across modern cloud platforms including ...

happy for you to learn Worked at a cutting edge tech company – Think FAANG level companies or similar Exposure to Data Engineering and Batch Processing Systems is highly advantageous They are paying between £140,000 and £180,000 with up to a 15% bonus and shares available. ...

Guidewire xCenters (version 10.x & SaaS) customization and integration (Intra and Inter system integration) using REST and SOAP APIs • Technical Skills: o Experience with batch processes, Guidewire Messaging framework, and OOTB plugins o Familiarity with Infra concepts and deployment architecture (WMB, WAS, and Apache) o Strong knowledge on Software architecture ...

understanding of .NET memory management internals - GC generations, LOH, pinned objects, finalisation, and how allocation patterns affect runtime behaviour Experience optimising high-throughput data processing pipelines - including async/await patterns, Channel, pipelines, and avoiding synchronisation bottlenecks Proven ability to diagnose large dataset query performance issues across both relational … patterns, LOH fragmentation, finaliser queues, and GC tuning (gen0/1/2, server vs. Workstation, DATAS) Investigate and optimise high-throughput data pipelines - batch processing, streaming patterns, buffer management, and backpressure handling Identify and resolve large dataset query performance issues - index utilisation, query plan analysis, pagination strategies ...