1 to 25 of 68 Batch Processing Jobs in London

dynamic team. The ideal candidate will have a strong background in Java development, extensive experience with AWS, database management, and preferred work experience in batch processing. As a Senior Java Developer, you will play a crucial role in designing, developing, and maintaining high-performance applications that meet our business … deploy cloud-based solutions, ensuring high availability and scalability.Database Management: Design, implement, and maintain database schemas, write complex SQL queries, and optimize database performance.Batch Processing: Develop and manage batch processing systems to handle large volumes of data efficiently.Code Quality: Ensure code quality through code reviews, unit testing ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data.Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark.Orchestration & Workflow: Build automated data ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

requirements are met. Mainframe Engineering: Support IBM z/OS infrastructure and associated mainframe technologies. Work with JCL, TSO/ISPF, SDSF, CICS, and batch processing environments. Analyse system and application performance using available monitoring and diagnostic tools. Participate in disaster recovery testing and business continuity exercises. Collaborate … requirements. Mainframe Technologies: Extensive experience in IBM z/OS environments. Strong working knowledge of: JCL TSO/ISPF SDSF CICS VSAM RACF Mainframe batch processing Familiarity with system monitoring and diagnostic tools. Development & Engineering: Experience supporting applications developed in COBOL. Good understanding of relational database concepts ...

requirements are met. Mainframe Engineering: Support IBM z/OS infrastructure and associated Mainframe technologies. Work with JCL, TSO/ISPF, SDSF, CICS, and batch processing environments. Analyse system and application performance using available monitoring and diagnostic tools. Participate in disaster recovery testing and business continuity exercises. Collaborate … requirements. Mainframe Technologies: Extensive experience in IBM z/OS environments. Strong working knowledge of: JCL TSO/ISPF SDSF CICS VSAM RACF Mainframe batch processing Familiarity with system monitoring and diagnostic tools. Development & Engineering: Experience supporting applications developed in Cobol. Good understanding of relational database concepts ...

About The Role FDM is a global business and technology consultancy seeking a DevOps Engineer (Batch Monitoring) to work for our client within the Banking & Financial Services sector. This is initially a 12 month contract with the potential to extend and will be a hybrid role that will … management, platform stability, automation, and continuous improvement to ensure key reporting processes are delivered reliably and efficiently. Responsibilities Monitor the health and performance of batch processing, applications, infrastructure, and data pipelines across the SFRC platform Perform first line incident management, root cause analysis, and operational troubleshooting activities Coordinate ...

comfortable taking ownership of architecture, engineering standards and complex technical decisions. Responsibilities: Lead technical design and delivery across the full SDLC. Build reliable batch-processing services for market and historical data. Design durable, searchable data stores and publish trusted datasets downstream. Ensure risk outputs are traceable and reproducible. … operations and an on-call rota when required. Set engineering standards, review designs and drive technical decisions. Key Skills: Strong experience building data-intensive, batch-processing platforms, ideally using market or time-series data. Advanced Java 11+ development, with Java 17+ preferred. Strong Spring/Spring Boot experience. ...

market data, reference data, pricing data, data quality, data lineage, data controls, and integration workflows. Mandatory experience analysing and documenting data pipelines, APIs, batch processes, event-driven workflows, databases, file-based interfaces, or cloud-based data platforms. Ability to write clear functional specifications, data mappings, user stories, acceptance criteria ...

understanding of market data, reference data, pricing data, data quality, data lineage, data controls, and integration workflows.Mandatory experience analysing and documenting data pipelines, APIs, batch processes, event-driven workflows, databases, file-based interfaces, or cloud-based data platforms.Ability to write clear functional specifications, data mappings, user stories, acceptance criteria ...

market data, reference data, pricing data, data quality, data lineage, data controls, and integration workflows. Mandatory experience analysing and documenting data pipelines, APIs, batch processes, event-driven workflows, databases, file-based interfaces, or cloud-based data platforms. Ability to write clear functional specifications, data mappings, user stories, acceptance criteria ...

market data, reference data, pricing data, data quality, data lineage, data controls, and integration workflows Mandatory experience analyzing and documenting data pipelines, APIs, batch processes, event-driven workflows, databases, file-based interfaces, or cloud-based data platforms Ability to write clear functional specifications, data mappings, user stories, acceptance criteria ...

collaborate more effectively, work smarter, and produce high-quality research more efficiently.**Key responsibilities:*** Design, build, and maintain scalable data pipelines and services* Develop batch and ELT/ETL data workflows for ingesting and processing large datasets* Work with technologies such as Python, Java, Spark, SQL, and Airflow … Strong skills in Python and experience with at least one backend language (such as Java)* Understanding of data engineering concepts such as data lakes, batch processing, and distributed systems* Experience building or operating data pipelines using tools like Spark and Airflow* Familiarity with cloud platforms (AWS preferred ...

Experiment with open source tools Participate in community-building initiatives such as meetups Continuously learn and improve engineering practices What we’re looking for batch processing distributed data processing Experience with technologies such as: Kafka Flink Spark Databricks experience is a plus Experience implementing: Data Lakes ...

them to collaborate more effectively, work smarter, and produce high-quality research more efficiently.Key responsibilities:Design, build, and maintain scalable data pipelines and servicesDevelop batch and ELT/ETL data workflows for ingesting and processing large datasetsWork with technologies such as Python, Java, Spark, SQL, and AirflowApply good … systemsStrong skills in Python and experience with at least one backend language (such as Java)Understanding of data engineering concepts such as data lakes, batch processing, and distributed systemsExperience building or operating data pipelines using tools like Spark and AirflowFamiliarity with cloud platforms (AWS preferred) and production monitoringClear ...

data into analytics repositories or feature stores, guaranteeing that AI models have consistent, well-prepared data to work with. This includes setting up stream processing for real-time data when required and automating pipeline orchestration for efficiency.Data Governance & Quality: Establish and enforce data governance policies and standards. This means … custom scripting with Python/SQL) to gather and transform data. You should understand how to optimise data flow and have experience with batch processing; experience with real-time streaming data (e.g. using Kafka or equivalent) is a plus.Ontologies & Knowledge Graphs: Exposure to semantic data modelling, ontologies ...

implementation of data quality, monitoring, observability, operational controls, and governance requirements. Lead the delivery of scalable data integration solutions across APIs, event-driven architectures, batch processing, and file-based integration patterns. Establish and govern engineering practices including CI/CD, automated testing, version control, infrastructure-as-code ...

implementation of data quality, monitoring, observability, operational controls, and governance requirements. Lead the delivery of scalable data integration solutions across APIs, event‐driven architectures, batch processing, and file‐based integration patterns. Establish and govern engineering practices including CI/CD, automated testing, version control, infrastructure‐as‐code ...

systems while preserving provenance and auditability. Build APIs and services that make processed data available to products, customers and AI systems. Develop workflows for batch processing, event-driven processing and long-running computational jobs. Create monitoring and alerting for pipeline failures, unexpected data changes and quality regressions. ...

cutting edge What we’re looking for Software & Data Engineering Strong expertise in Python Passion for writing: clean reliable production-grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience ...

metadata-forward, CI/CD-driven platform represents and enables the entire application and analysis lifecycle including interactive development and explorations (notebooks), large-scale batch processing, observability and production application deployments. A Sr. Compute Platform Engineer is a leading technical contributor who can consistently take a poorly defined … business problems, Responsible for development of key components of a hybrid on-prem/cloud compute platform for both interactive and scalable batch computing and establishing of processes and workflows to transition existing HPC users and teams to this platform, Responsible for code-driven environment, applications, and container/ ...

native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such as Glue, EMR Serverless … experience writing clean, maintainable Python code, ideally using type hints, linters, and test frameworks such as pytest. Solid understanding of data engineering fundamentals including batch processing, schema evolution, and ETL pipeline development. Experience with—or strong interest in learning—Apache Spark for large-scale data processing. Familiarity with ...

with performance or load testing tools (JMeter, Gatling). Familiarity with Kubernetes, Terraform, or Infrastructure as Code. Exposure to data validation, ETL pipelines, or batch processing systems. At the company, we believe diverse perspectives lead to better outcomes, and inclusion creates the conditions for everyone to thrive. ...

team WHAT YOU'LL BRING: Writes clean, maintainable Python code (ideally with type hints, linters, and tests like pytest) Understands data engineering basics: batch processing, schema evolution, and building ETL pipelines Has experience with or is eager to learn Apache Spark for large-scale data processing ...

placement, policy, claims, exposure, bordereaux, and risk analytics. Strong knowledge of data integration patterns including ETL/ELT, APIs, event-driven architecture, and batch processing. Experience with modern cloud-based data platforms and distributed data architectures. Experience with both relational and non-relational data platforms , including exposure to MongoDB ...