1 to 25 of 28 Batch Processing Jobs in England

analytics capabilities across complex secure systems. The successful candidate will play a key role in architecting resilient data platforms, optimising real-time and batch processing pipelines, and ensuring compliance with strict governance and security standards. KEY RESPONSIBILITIES Data Architecture & Engineering - Design and implement scalable, resilient data pipeline architectures … using Elastic Stack and Apache NiFi - Develop secure frameworks for data ingestion, transformation and integration - Build and maintain batch and real-time data processing pipelines - Ensure high standards of data quality, integrity and traceability across all systems Elastic Stack & Platform Management - Configure, optimise and maintain Elasticsearch clusters ...

Design Implement enterprise-level features such as: Dependency Injection/Inversion of Control (IoC) Aspect-Oriented Programming (AOP) Functional programming patterns Transaction management frameworks Batch processing solutions Participate in systems design and architecture discussions Support systems integration across enterprise platforms and services DevOps & Delivery Apply DevOps methodologies …/JEE and web technologies Knowledge of: Dependency Injection (e.g., Spring) AOP and IoC Data access frameworks (e.g., Hibernate/JPA) Transaction management systems Batch processing frameworks Familiarity with DevOps tools and practices Understanding of: Information security principles System integration and APIs Modern software engineering standards Key Competencies ...

experience AWS Azure Desirable Skills: Hands-on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large-scale batch processing. Data engineering at scale in government or similarly complex domains, including performance tuning and data quality management. CI/CD & DevOps: pipelines ...

modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. API integration and data services. Data ...

strong plus. Your responsibilities: Architect and implement scalable, secure, and high-performance data platforms using Azure, Databricks, and Microsoft Fabric for Real Time and batch processing. Lead integration across structured and unstructured data sources such as SCADA, SAP, APIs, telemetry, and time-series data using modern ETL/ ...

Microsoft Fabric. Your responsibilities: Architect and implement scalable, secure, and high-performance data platforms using Azure, Databricks, and Microsoft Fabric for real-time and batch processing. Lead integration across structured and unstructured data sources such as SCADA, SAP, APIs, telemetry, and time-series data using modern ETL/ ...

experienced AWS Data Engineer to help design, build, and maintain a secure, scalable data platform for a government client - supporting real-time and batch processing that underpins critical analytics and decision-making. What you'll be doing: Building, testing, and maintaining robust data pipelines Designing logical and physical … data models Scaling the data platform to meet evolving use cases Analysing data flows and improving processing performance Ensuring data architecture aligns with agreed standards Troubleshooting issues and driving continuous improvement What we're looking for: Essential: Proven experience building data pipelines Strong understanding of data modelling and structures ...

with observability frameworks (Prometheus, ElasticSearch , Grafana, OpenTelemetry ) for metrics, logs and traces Platform Technologies & BSS Vendor Platforms - Open source workflow engines (Temporal) - Stream/batch processing (Flink, Spark) - Test automation: Robot Framework and BDD - can review BDD robot files - BSS vendor platforms: Salesforce, Amdocs, Ciena - BSS/ ...

experience working in Data Warehouse/ETL environments Experience in data integration, transformation, and performance tuning Strong understanding of ETL frameworks, data pipelines, and batch processing Desirable Skills Working knowledge of SAS (especially in ETL context) Experience in migrating SAS workloads to Informatica ETLs Familiarity with data modelling ...

priorities and ensuring high-quality delivery. Design, build, and maintain scalable OT-to-business data architectures and ingestion pipelines, supporting both real-time and batch processing. Integrate operational technology platforms, including AVEVA PI, with ERP, maintenance, quality, and other enterprise systems. Develop and maintain standardised, granular datasets and data ...

leadership and mentorship to cross-functional data teams Define best practices across data architecture, cloud engineering, and security Design and oversee real-time and batch processing solutions, including streaming architectures Ensure compliance with government standards, security policies, and data regulations Monitor delivery against architectural designs and manage ...

function, setting standards, mentoring engineers, and working closely with both internal teams and external delivery partners. The Role Data Engineering & Pipeline Development Transform legacy batch processes into scalable, observable data pipelines Build and deliver high-quality data products in line with architectural designs Work across modern cloud platforms including ...

projects. Knowledge of Azure, AWS or GCP environments would be an advantage. Oracle SQL/PLSQL and data modelling expertise. SITS configuration, workflows, security, batch processing, and role based access. Integration experience using StuTalk, APIs, middleware, and data feeds . Agile delivery experience within complex enterprise environments. Higher ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

with engineering standards and change control processes Support and enhance system functionality to ensure accuracy, resilience and performance—especially around stock integrity and sales processing Diagnose and resolve complex technical issues, providing 3rd line support and identifying long-term fixes Deliver system changes, upgrades and new store rollouts with … Hands-on experience with COBOL and Java, or a strong willingness to work across both Experience working in a Linux environment, including scripting and batch processing Knowledge of transactional or back-office systems (e.g. stock control, sales processing, financial systems), ideally in retail or wholesale Strong problem ...

dataset publishing* Build and maintain FastAPI backends, vanilla JavaScript frontends, CLI tooling, and supporting services* Improve recording workflows, state machines, config validation, batch processing, stale-data recovery, and archival processes* Containerise and orchestrate services to ensure reliable deployment across machines and lab environments* Partner with ML engineers … pipelines, distributed systems, or performance-sensitive IO workloads* Experience orchestrating external CLI tools from Python via subprocess, ideally ffmpeg or similar data/media processing tools Nice to Have It would be useful to have: * Distributed ML training infrastructure experience, such as PyTorch DDP, FSDP, DeepSpeed, sharded datasets, streaming ...

application incidents related to database logic, Oracle Forms, Reports, and WebLogic deployments. Conduct root cause analysis for recurring issues and implement permanent fixes. Support batch processes, scheduled jobs, and integration feeds to ensure business continuity. Work closely with business users to analyse defects and translate requirements into system enhancements. ...

highly desirable. Strong capability across Salesforce Lightning, Flow automation, and security configuration is expected, along with familiarity in Apex development (including triggers and batch processing) and Lightning Web Components. Experience with Omnistudio (including Omniscripts, Flex Cards, DataRaptors, and Integration Procedures) will be highly advantageous, particularly in enterprise-scale ...

operating in complex, fast paced transformation environments Proactive, resilient, and delivery focused Deep understanding of integration architecture and patterns, including: APIs File-based interfaces Batch processing Data transformations Hands on expertise in data modelling, mapping, and reconciliation Experience working across complex, multi-system landscapes Ability to lead solution ...

systems where stability is critical. Data Savvy: Experience working with data pipelines and a solid grasp of the trade-offs between real-time and batch processing. Mindset: An eye for usability. You know how to make high-stakes, complex workflows feel simple and foolproof for non-technical users. ...

join us. What skills, knowledge and experience will you need? Experience bringing together data from lots of different systems, including real-time and batch processing Strong understanding of ETL/ELT tools and how data is extracted, transformed and loaded into platforms Knowledge of data warehousing, data lakes … operate across the entire Data & Analytics architecture, focusing on assurance and coherence. Our data landscape can be thought of as layered - covering ingestion, storage, processing and consumption - but users experience seamless services end-to-end. Your role is to ensure those layers work together as a joined-up whole ...

join us. What skills, knowledge and experience will you need? Experience bringing together data from lots of different systems, including real-time and batch processing Strong understanding of ETL/ELT tools and how data is extracted, transformed and loaded into platforms Knowledge of data warehousing, data lakes … operate across the entire Data & Analytics architecture, focusing on assurance and coherence. Our data landscape can be thought of as layered - covering ingestion, storage, processing and consumption - but users experience seamless services end-to-end. Your role is to ensure those layers work together as a joined-up whole ...

join us. What skills, knowledge and experience will you need? Experience bringing together data from lots of different systems, including real-time and batch processing Strong understanding of ETL/ELT tools and how data is extracted, transformed and loaded into platforms Knowledge of data warehousing, data lakes … operate across the entire Data & Analytics architecture, focusing on assurance and coherence. Our data landscape can be thought of as layered - covering ingestion, storage, processing and consumption - but users experience seamless services end-to-end. Your role is to ensure those layers work together as a joined-up whole ...

join us. What skills, knowledge and experience will you need? Experience bringing together data from lots of different systems, including real-time and batch processing Strong understanding of ETL/ELT tools and how data is extracted, transformed and loaded into platforms Knowledge of data warehousing, data lakes … operate across the entire Data & Analytics architecture, focusing on assurance and coherence. Our data landscape can be thought of as layered - covering ingestion, storage, processing and consumption - but users experience seamless services end-to-end. Your role is to ensure those layers work together as a joined-up whole ...