1 to 25 of 64 Batch Processing Jobs in London

dynamic team. The ideal candidate will have a strong background in Java development, extensive experience with AWS, database management, and preferred work experience in batch processing. As a Senior Java Developer, you will play a crucial role in designing, developing, and maintaining high-performance applications that meet our business … cloud-based solutions, ensuring high availability and scalability. Database Management: Design, implement, and maintain database schemas, write complex SQL queries, and optimize database performance. Batch Processing: Develop and manage batch processing systems to handle large volumes of data efficiently. Code Quality: Ensure code quality through code ...

business analysis and production support groups, ensuring platform improvements translate into stronger production stability, better compute utilisation, more reliable lower-lane execution, improved batch observability, data quality controls, disciplined regression and release processes, and sustained adoption of agreed engineering changes. Role Description: This is a critical technical platform leadership … FRTB Standardised Approach. This is not a programme management, advisory or coordination role. You should be technically credible, able to understand platform architecture, batch execution, data flows, dependencies, production failure modes, hardware utilisation, computation cost and engineering trade-offs, and capable of challenging technical assumptions. We require an execution ...

Design, develop, test, deploy, and maintain scalable ETL/ELT pipelines for structured and unstructured data. Build batch and real-time processing solutions using BigQuery, Dataflow, Cloud Storage, Pub/Sub, and related GCP services. Develop data-processing components and integrations using Java or Python, SQL, REST … transformation frameworks supporting full, incremental, and event-driven loads. Design data models and optimize BigQuery tables through partitioning, clustering, efficient SQL, and cost-aware processing patterns. Produce high-level and low-level technical designs, document assumptions, and contribute to architecture reviews. Implement data-quality checks, schema validation, reconciliation, audit ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

enterprise data platform. In this role, you will implement a modern data lakehouse architecture capable of supporting raw and curated data zones, scalable batch processing, and future analytical workloads. You will play a key role in integrating our existing data pipelines, models, and transformation logic into a unified … Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated ...

enterprise data, APIs and microservices. Ensure AI solutions integrate cleanly into cloud-native PaaS and SaaS environments, with robust approaches to real-time and batch processing where required. Establish best practices for prompt engineering, model selection, grounding, evaluation and guardrails across AI solutions. Lead AI and LLM proof … guardrails. Experience working with cloud environments and enterprise platforms, with Azure experience preferred, and an understanding of PaaS, SaaS, real-time and batch processing patterns. Familiarity with Docker, CI/CD, infrastructure approaches, MLOps, LLMOps, AIOps, observability, versioning and production operations would be advantageous. Strong understanding of responsible ...

high-quality data products everywhere; and we directly own a deliberate set of business-critical, cross-cutting data products. Our work spans: Streaming and batch data platforms that unlock data across TripadvisorETL pipelines and data models serving terabytes and growing datasets in Snowflake, BigQuery, and HiveOrchestration, scheduling …/ELT data pipelines that process terabyte- to petabyte-scale data across Snowflake, BigQuery, and other platforms. Design, build and maintain streaming and batch systems that power real-time and analytical workloads, and lead simplification of legacy ingestion paths. Drive warehouse standardization and technical-debt reduction as we consolidate ...

data Lakehouse solutions using Databricks, Snowflake, and cloud-native servicesDesign high-performance data models, ingestion frameworks, and transformation pipelines supporting both real-time and batch processing workloadsEstablish best practices for data governance, lineage, security, metadata management, and platform observabilityBuild and optimise analytical platforms capable of processing large ...

Lakehouse solutions using Databricks, Snowflake, and cloud-native services Design high-performance data models, ingestion frameworks, and transformation pipelines supporting both real-time and batch processing workloads Establish best practices for data governance, lineage, security, metadata management, and platform observability Build and optimise analytical platforms capable of processing ...

learning techniques, including supervised ML, unsupervised ML, and LLMs Experience with deployment tools and processes, such as Azure Data Factory (ADF), Azure ML and batch processing Experience using DevOps or equivalent agile development environments 4+ years’ experience leading clients through complex technical projects — you know how to navigate ...

Experiment with open source tools Participate in community-building initiatives such as meetups Continuously learn and improve engineering practices What we’re looking for batch processing distributed data processing Experience with technologies such as: Kafka Flink Spark Databricks experience is a plus Experience implementing: Data Lakes ...

details, including source-to-target data mappings, message and file transformations, system-to-system interactions, interface contracts, validation rules, exception paths, and processing logic. The analyst will help a non-technical Product Owner convert business outcomes and priorities into technically precise requirements that engineering teams can design, build … source-to-target mappings, field-level specifications, message structures, and validation requirements. Document end-to-end system interactions, including upstream and downstream dependencies, interfaces, processing sequences, handoffs, exception scenarios, and operational impacts. Review existing application behavior, technical documentation, schemas, specifications, logs, and, where necessary, pseudo code or logic representations ...

build the services and infrastructure behind portfolio risk and analytics calculations. You’ll turn analytical models into reliable production services, building APIs, cloud processing and workflows that connect portfolio data with calculation engines and deliver results to existing systems. There is also scope to provide hands‐on technical leadership … across the wider team. The role Build Python APIs and calculation services. Develop cloud-based processing for scheduled batch and on‐demand jobs. Implement containers, orchestration and queues, with robust retries and recovery. Integrate SQL data sources and downstream systems, ensuring accurate, traceable inputs and outputs. Build deployment ...

systems while preserving provenance and auditability. Build APIs and services that make processed data available to products, customers and AI systems. Develop workflows for batch processing, event-driven processing and long-running computational jobs. Create monitoring and alerting for pipeline failures, unexpected data changes and quality regressions. ...

cutting edge What we’re looking for Software & Data Engineering Strong expertise in Python Passion for writing: clean reliable production-grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience ...

native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such as Glue, EMR Serverless … experience writing clean, maintainable Python code, ideally using type hints, linters, and test frameworks such as pytest. Solid understanding of data engineering fundamentals including batch processing, schema evolution, and ETL pipeline development. Experience with—or strong interest in learning—Apache Spark for large-scale data processing. Familiarity with ...

with performance or load testing tools (JMeter, Gatling). Familiarity with Kubernetes, Terraform, or Infrastructure as Code. Exposure to data validation, ETL pipelines, or batch processing systems. At the company, we believe diverse perspectives lead to better outcomes, and inclusion creates the conditions for everyone to thrive. ...

tune enterprise-grade ETL pipelines (using tools like Informatica, Talend, or modern cloud data engines) to extract, cleanse, and transform derivative lifecycle data Product Processing Mapping: Map complex clearing workflows, including trade capture, position allocations, give-up/give-in workflows, exchange fee schedules, and margins across 40+ global … closely with Business Analysts, Risk, Product Control, and Operations teams to translate functional specifications into clean, production-grade ETL code Performance Tuning : Optimize massive batch and real-time streaming data loads (using Autosys, Control-M, or Kafka) to comfortably fit within tight overnight batch processing windows ...

across various operational processes. Key Responsibilities: Design, implement, and execute data quality controls to uphold the completeness, accuracy, and integrity of reporting data across batch runs. Execute validation checks for daily batch processing cycles, ensuring adherence to established data quality standards. Conduct reconciliations between upstream, downstream ...

complex enterprise application landscapes undergoing legacy transformation. You will focus on shaping scalable, resilient, secure, and governable architectures that progressively move monolithic, tightly coupled, batch-oriented systems toward modular, event-driven, cloud-enabled and operationally observable platforms. The role requires strong hands‐on architecture depth across application modernization, integration … modernization, distributed systems, integration platforms, database ecosystems, cloud-enabled platforms, and regulated enterprise environments. Strong ability to assess monolithic, tightly coupled, database-centric and batch-driven estates and translate findings into practical target-state architecture and migration roadmaps. Deep understanding of microservices architecture, domain-driven design, service decomposition, database ...

Doing Designing and developing scalable Python-based services and APIs. Building cloud-native platforms supporting analytical and risk workloads. Developing orchestration frameworks for batch and on-demand calculations. Managing containerised environments and distributed compute services. Implementing monitoring, observability and operational tooling. Building resilient services with fault tolerance, recovery … monitoring, observability and operational support. Strong stakeholder engagement and communication skills. Desirable Experience Risk systems, portfolio analytics or financial services platforms. Large-scale batch processing or calculation-heavy environments. Model governance, version control and auditable execution frameworks. Leading engineering teams or mentoring technical colleagues. #J-18808-Ljbffr ...

Doing Designing and developing scalable Python-based services and APIs. Building cloud-native platforms supporting analytical and risk workloads. Developing orchestration frameworks for batch and on-demand calculations. Managing containerised environments and distributed compute services. Implementing monitoring, observability and operational tooling. Building resilient services with fault tolerance, recovery … monitoring, observability and operational support. Strong stakeholder engagement and communication skills. Desirable Experience Risk systems, portfolio analytics or financial services platforms. Large-scale batch processing or calculation-heavy environments. Model governance, version control and auditable execution frameworks. Leading engineering teams or mentoring technical colleagues. Why Apply? This ...

improvements Supporting D365 configuration, workflows, security, user administration and functional troubleshooting Investigating incidents and system issues, identifying root causes and coordinating resolutions Supporting integrations, batch processes, data interfaces and data management Planning and supporting UAT, regression and validation testing, including test scripts, defect management and stakeholder sign-off Supporting … hands-on approach and the ability to take ownership of issues through to resolution Knowledge of D365 workflows, security administration, LCS, DMF, integrations and batch processing would be particularly useful. Exposure to Power BI, Power Automate, Power Apps, SQL or other ERP platforms would also be advantageous. ...

improvements Supporting D365 configuration, workflows, security, user administration and functional troubleshooting Investigating incidents and system issues, identifying root causes and coordinating resolutions Supporting integrations, batch processes, data interfaces and data management Planning and supporting UAT, regression and validation testing, including test scripts, defect management and stakeholder sign-off Supporting … hands-on approach and the ability to take ownership of issues through to resolution Knowledge of D365 workflows, security administration, LCS, DMF, integrations and batch processing would be particularly useful. Exposure to Power BI, Power Automate, Power Apps, SQL or other ERP platforms would also be advantageous. ...

Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing both batch processing and real-time streaming architectures. Expertise in data warehousing concepts, advanced ETL/ELT processes, and analytics engineering best practices. Exceptional programming ...