1 to 25 of 45 Batch Processing Jobs in London

stability.**Technical Responsibilities*** Monitor and support applications, infrastructure, databases, middleware, integrations, and cloud services across hybrid environments.* Triage application, database, server, network, middleware, integration, batch-processing, and cloud-service issues.* Support controlled deployments, validation, environment readiness, and production release activities.* Automate support, deployment, validation, monitoring, and documentation activities … using approved DevOps and scripting tools.* Analyze SQL and Oracle data, logs, APIs, middleware, batch processing, and data pipelines to restore service and support root-cause analysis.* Support operational readiness, backup, recovery, resiliency, availability, performance, and disaster-recovery activities.**Success Profile**To thrive in this role ...

Design, develop, test, deploy, and maintain scalable ETL/ELT pipelines for structured and unstructured data. Build batch and real-time processing solutions using BigQuery, Dataflow, Cloud Storage, Pub/Sub, and related GCP services. Develop data-processing components and integrations using Java or Python, SQL, REST … transformation frameworks supporting full, incremental, and event-driven loads. Design data models and optimize BigQuery tables through partitioning, clustering, efficient SQL, and cost-aware processing patterns. Produce high-level and low-level technical designs, document assumptions, and contribute to architecture reviews. Implement data-quality checks, schema validation, reconciliation, audit ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

enterprise data, APIs and microservices. Ensure AI solutions integrate cleanly into cloud-native PaaS and SaaS environments, with robust approaches to real-time and batch processing where required. Establish best practices for prompt engineering, model selection, grounding, evaluation and guardrails across AI solutions. Lead AI and LLM proof … guardrails. Experience working with cloud environments and enterprise platforms, with Azure experience preferred, and an understanding of PaaS, SaaS, real-time and batch processing patterns. Familiarity with Docker, CI/CD, infrastructure approaches, MLOps, LLMOps, AIOps, observability, versioning and production operations would be advantageous. Strong understanding of responsible ...

This involves working across the data lake, reporting systems, and downstream feeds to ensure data integrity and smooth transitions. Key responsibilities include large-scale batch processing, meticulous data validation, and comprehensive regression testing. You will also be instrumental in preparing for migration readiness activities within a dynamic financial … processes are robust and reliable. Requirements Proven experience as a Data Engineer or Data Analyst, particularly within financial services. Strong background in large-scale batch processing and data validation. Experience with regression testing and migration readiness activities. Proficiency in data extraction, transformation, reconciliation, and validation. Excellent problem-solving ...

meticulous work across the data lake, reporting systems, and downstream feeds, ensuring data integrity and operational efficiency. Key responsibilities include engaging in large-scale batch processing, robust data validation, thorough regression testing, and preparing for migration readiness within a dynamic financial services environment. You will be instrumental … success of critical projects. Requirements Proven experience as a Data Engineer or Data Analyst, particularly within financial services. Strong background in large-scale batch processing and data validation. Familiarity with ledger migration activities, including data extraction, transformation, reconciliation, and validation. Ability to perform regression testing and support migration ...

learning techniques, including supervised ML, unsupervised ML, and LLMs Experience with deployment tools and processes, such as Azure Data Factory (ADF), Azure ML and batch processing Experience using DevOps or equivalent agile development environments 4+ years’ experience leading clients through complex technical projects — you know how to navigate ...

implementation of data quality, monitoring, observability, operational controls, and governance requirements. Lead the delivery of scalable data integration solutions across APIs, event‐driven architectures, batch processing, and file‐based integration patterns. Establish and govern engineering practices including CI/CD, automated testing, version control, infrastructure‐as‐code ...

Experiment with open source tools Participate in community-building initiatives such as meetups Continuously learn and improve engineering practices What we’re looking for batch processing distributed data processing Experience with technologies such as: Kafka Flink Spark Databricks experience is a plus Experience implementing: Data Lakes ...

systems while preserving provenance and auditability. Build APIs and services that make processed data available to products, customers and AI systems. Develop workflows for batch processing, event-driven processing and long-running computational jobs. Create monitoring and alerting for pipeline failures, unexpected data changes and quality regressions. ...

cutting edge What we’re looking for Software & Data Engineering Strong expertise in Python Passion for writing: clean reliable production-grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience ...

native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such as Glue, EMR Serverless … experience writing clean, maintainable Python code, ideally using type hints, linters, and test frameworks such as pytest. Solid understanding of data engineering fundamentals including batch processing, schema evolution, and ETL pipeline development. Experience with—or strong interest in learning—Apache Spark for large-scale data processing. Familiarity with ...

with performance or load testing tools (JMeter, Gatling). Familiarity with Kubernetes, Terraform, or Infrastructure as Code. Exposure to data validation, ETL pipelines, or batch processing systems. At the company, we believe diverse perspectives lead to better outcomes, and inclusion creates the conditions for everyone to thrive. ...

tune enterprise-grade ETL pipelines (using tools like Informatica, Talend, or modern cloud data engines) to extract, cleanse, and transform derivative lifecycle data Product Processing Mapping: Map complex clearing workflows, including trade capture, position allocations, give-up/give-in workflows, exchange fee schedules, and margins across 40+ global … closely with Business Analysts, Risk, Product Control, and Operations teams to translate functional specifications into clean, production-grade ETL code Performance Tuning : Optimize massive batch and real-time streaming data loads (using Autosys, Control-M, or Kafka) to comfortably fit within tight overnight batch processing windows ...

complex enterprise application landscapes undergoing legacy transformation. You will focus on shaping scalable, resilient, secure, and governable architectures that progressively move monolithic, tightly coupled, batch-oriented systems toward modular, event-driven, cloud-enabled and operationally observable platforms. The role requires strong hands‐on architecture depth across application modernization, integration … modernization, distributed systems, integration platforms, database ecosystems, cloud-enabled platforms, and regulated enterprise environments. Strong ability to assess monolithic, tightly coupled, database-centric and batch-driven estates and translate findings into practical target-state architecture and migration roadmaps. Deep understanding of microservices architecture, domain-driven design, service decomposition, database ...

across various operational processes. Key Responsibilities: Design, implement, and execute data quality controls to uphold the completeness, accuracy, and integrity of reporting data across batch runs. Execute validation checks for daily batch processing cycles, ensuring adherence to established data quality standards. Conduct reconciliations between upstream, downstream ...

Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing both batch processing and real-time streaming architectures. Expertise in data warehousing concepts, advanced ETL/ELT processes, and analytics engineering best practices. Exceptional programming ...

placement, policy, claims, exposure, bordereaux, and risk analytics. Strong knowledge of data integration patterns including ETL/ELT, APIs, event-driven architecture, and batch processing. Experience with modern cloud-based data platforms and distributed data architectures. Experience with both relational and non-relational data platforms , including exposure to MongoDB ...

Guidewire xCenters (version 10.x & Cloud/SaaS) customization and integration (Intra and Inter system integration) using REST and SOAP APIs. Technical Skills: Experience with batch processes, Guidewire Messaging framework and OOTB plugins; familiarity with infra concepts and deployment architecture (WMB, WAS and Apache); strong knowledge of software architecture ...

experience of developing complex solutions in a corporate environment. Solid understanding of working with production codebases, such as version control, CI/CD, and batch processing. Practical experience working on cloud-based ML platforms. Development and coaching A strong awareness and understanding of technology trends and direction in Data ...

trading activities. Take full ownership of data products, guiding them from initial concept through to stable production. Design, build, and maintain systems for both batch processing and real-time streaming of time series datasets, ensuring high data quality and reliability. Develop APIs and data access methods for fast … databases Have a Bachelor or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information about DRW's processing activities and our use of job applicants' data, please view our Privacy Notice at https://drw.com/privacy-notice #J-18808-Ljbffr ...

trading activities. Take full ownership of data products, guiding them from initial concept through to stable production. Design, build, and maintain systems for both batch processing and real‐time streaming of time series datasets, ensuring high data quality and reliability. Develop APIs and data access methods for fast … databases Have a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information about DRW’s processing activities and our use of job applicants’ data, please view our Privacy Notice at https://drw.com/privacy-notice. California residents, please ...

maintain reports using Endur reporting tools Support CI/CD processes and deployments within Endur environments Troubleshoot issues across development, testing, and production Support batch processes and EOD activities as needed Work with business users and analysts to understand requirements and deliver working solutions Required (Endur-specific) Hands … working within Endur reporting tools (Report Builder or similar) Familiarity with CI/CD processes in Endur environments Experience supporting or working with Endur batch/EOD processes 3–5+ years of Endur‐specific development experience Transferable experience (CTRM/general development) Development experience with Java, JavaScript, .NET ...

next generation data volumes. You'll have the opportunity to help shape foundational infrastructure used across the company, supporting high-priority data runs, batch inference, evaluations and business-critical incident response. Key responsibilities Design, build and evolve scalable data pipeline and orchestration infrastructure for Wayve's next-generation autonomous … revamp of the existing Harness system and the development of a new DAG-based workflow platform Build tooling and libraries that make data ingestion, processing, inference and evaluation workflows simple, reliable and consistent across teams Ensure the platform can run hundreds of parallel pipeline steps efficiently, with correct outputs ...