15 of 15 Batch Processing Jobs in London

material changes. • Lifecycle events - platforms supporting the generation and reporting of trade lifecycle events, including payments, barrier and early termination events. • Market Risk Batch - batch processing services that generate risk measures, sensitivities, scenario outputs and reporting datasets used for risk oversight, control, analytics and downstream consumption, with ...

modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. API integration and data services. Data ...

collaborate more effectively, work smarter, and produce high-quality research more efficiently. Key responsibilities: Design, build, and maintain scalable data pipelines and services Develop batch and ELT/ETL data workflows for ingesting and processing large datasets Work with technologies such as Python, Java, Spark, SQL, and Airflow … Strong skills in Python and experience with at least one backend language (such as Java) Understanding of data engineering concepts such as data lakes, batch processing, and distributed systems Experience building or operating data pipelines using tools like Spark and Airflow Familiarity with cloud platforms (AWS preferred ...

metadata-forward, CI/CD-driven platform represents and enables the entire application and analysis lifecycle including interactive development and explorations (notebooks), large-scale batch processing, observability and production application deployments. A Compute Platform Engineer II is a technical contributor who can consistently take a poorly defined business … business problems, Responsible for development of key components of a hybrid on-prem/cloud compute platform for both interactive and scalable batch computing and establishing of processes and workflows to transition existing HPC users and teams to this platform Responsible for code-driven environment, applications, and container/ ...

background in modern data platforms, cloud-native architecture and Databricks? We're supporting a major programme that is modernising and transforming large-scale data processing capabilities and are looking for a hands-on architect who can bridge strategy, design and delivery. Role This position will play a key role … technical implementation layers, helping to guide migration activity while remaining hands-on with infrastructure, automation and platform delivery. Key Responsibilities Design scalable data processing solutions using AWS, Databricks and associated cloud services. Lead the transition of existing data workloads and integrations onto a modern cloud-native platform. Define platform ...

background in modern data platforms, cloud-native architecture and Databricks? We're supporting a major programme that is modernising and transforming large-scale data processing capabilities and are looking for a hands-on architect who can bridge strategy, design and delivery. Role This position will play a key role … technical implementation layers, helping to guide migration activity while remaining hands-on with infrastructure, automation and platform delivery. Key Responsibilities Design scalable data processing solutions using AWS, Databricks and associated cloud services. Lead the transition of existing data workloads and integrations onto a modern cloud-native platform. Define platform ...

background in modern data platforms, cloud-native architecture and Databricks? We're supporting a major programme that is modernising and transforming large-scale data processing capabilities and are looking for a hands-on architect who can bridge strategy, design and delivery. Role This position will play a key role … technical implementation layers, helping to guide migration activity while remaining hands-on with infrastructure, automation and platform delivery. Key Responsibilities Design scalable data processing solutions using AWS, Databricks and associated cloud services. Lead the transition of existing data workloads and integrations onto a modern cloud-native platform. Define platform ...

background in modern data platforms, cloud-native architecture and Databricks? We're supporting a major programme that is modernising and transforming large-scale data processing capabilities and are looking for a hands-on architect who can bridge strategy, design and delivery. Role This position will play a key role … technical implementation layers, helping to guide migration activity while remaining hands-on with infrastructure, automation and platform delivery. Key Responsibilities Design scalable data processing solutions using AWS, Databricks and associated cloud services. Lead the transition of existing data workloads and integrations onto a modern cloud-native platform. Define platform ...

with observability frameworks (Prometheus, ElasticSearch , Grafana, OpenTelemetry ) for metrics, logs and traces Platform Technologies & BSS Vendor Platforms - Open source workflow engines (Temporal) - Stream/batch processing (Flink, Spark) - Test automation: Robot Framework and BDD - can review BDD robot files - BSS vendor platforms: Salesforce, Amdocs, Ciena - BSS/ ...

environments. Degree in Computer Science, Software Engineering or equivalent. Knowledge of on‐premise and cloud technology operation. Experience with large‐scale real‐time and batch process orchestration, optionally with third‐party engines. Understanding of cyber‐security standards, secure‐by‐design principles. Agile framework experience (Scrum, Kanban). Background ...

experience working in Data Warehouse/ETL environments Experience in data integration, transformation, and performance tuning Strong understanding of ETL frameworks, data pipelines, and batch processing Desirable Skills Working knowledge of SAS (especially in ETL context) Experience in migrating SAS workloads to Informatica ETLs Familiarity with data modelling ...

Agent Kit, MCP Toolbox for Databases) Familiarity with dbt for transformation and data modeling at scale Experience building or operating streaming data infrastructure alongside batch processing Background in compliance‐sensitive environments (SOX, regulatory reporting, audit systems) Published work, conference talks, or open‐source contributions in knowledge engineering, semantic ...

Agent Kit, MCP Toolbox for Databases)* Familiarity with dbt for transformation and data modeling at scale* Experience building or operating streaming data infrastructure alongside batch processing* Background in compliance-sensitive environments (SOX, regulatory reporting, audit systems)* Published work, conference talks, or open-source contributions in knowledge engineering, semantic ...

leadership and mentorship to cross-functional data teams Define best practices across data architecture, cloud engineering, and security Design and oversee real-time and batch processing solutions, including streaming architectures Ensure compliance with government standards, security policies, and data regulations Monitor delivery against architectural designs and manage ...

various internal & external stakeholders using secure authentication patterns. Your role will include choosing and implementing the appropriate technologies for scaling data access patterns, batch processing, and data streaming for soft real-time consumption while considering the unique domain knowledge of the client's business. As a senior collaborator ...

function, setting standards, mentoring engineers, and working closely with both internal teams and external delivery partners. The Role Data Engineering & Pipeline Development Transform legacy batch processes into scalable, observable data pipelines Build and deliver high-quality data products in line with architectural designs Work across modern cloud platforms including ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

Role Design and deliver APIs using API-led connectivity (System, Process, Experience layers) Build complex integration solutions including event-driven architectures, orchestration, streaming, and batch processing Lead development on key projects, ensuring high-quality engineering standards Produce reusable assets, modules, and DataWeave transformations Contribute to API specifications (RAML ...

highly desirable. Strong capability across Salesforce Lightning, Flow automation, and security configuration is expected, along with familiarity in Apex development (including triggers and batch processing) and Lightning Web Components. Experience with Omnistudio (including Omniscripts, Flex Cards, DataRaptors, and Integration Procedures) will be highly advantageous, particularly in enterprise-scale ...

systems where stability is critical. Data Savvy: Experience working with data pipelines and a solid grasp of the trade-offs between real-time and batch processing. Mindset: An eye for usability. You know how to make high-stakes, complex workflows feel simple and foolproof for non-technical users. ...

systems where stability is critical. Data Savvy: Experience working with data pipelines and a solid grasp of the trade-offs between real-time and batch processing. Mindset: An eye for usability. You know how to make high-stakes, complex workflows feel simple and foolproof for non-technical users. ...