1 to 25 of 91 Batch Processing Jobs in London

working with modern tools in a fast-moving, high-performance environment. Your responsibilities may include: Build and maintain scalable, efficient ETL/ELT pipelines for both real-time and batch processing. Integrate data from APIs, streaming platforms, and legacy systems, with a focus on data quality and reliability. Design and manage data storage solutions, including databases, warehouses, and lakes. … Leverage cloud-native services and distributed processing tools (e.g., Apache Flink, AWS Batch) to support large-scale data workloads. Operations & Tooling Monitor, troubleshoot, and optimize data pipelines to ensure performance and cost efficiency. Implement data governance, access controls, and security measures in line with best practices and regulatory standards. Develop observability and anomaly detection tools to support Tier … maintaining scalable ETL/ELT pipelines and data architectures. Hands-on expertise with cloud platforms (e.g., AWS) and cloud-native data services. Comfortable with big data tools and distributed processing frameworks such as Apache Flink or AWS Batch. Strong understanding of data governance, security, and best practices for data quality. Effective communicator with the ability to work across technical More ❯

Python, SKLearn, XGBoost, SparkML, etc. Experience with deep learning frameworks like TensorFlow or PyTorch. Knowledge of ML model deployment options (e.g., Azure Functions, FastAPI, Kubernetes) for real-time and batch processing. Experience with CI/CD pipelines (e.g., DevOps pipelines, Git actions). Knowledge of infrastructure as code (e.g., Terraform, ARM Template, Databricks Asset Bundles). Understanding of advanced … machine learning techniques, including graph-based processing, computer vision, natural language processing, and simulation modeling. Experience with generative AI and LLMs, such as LLamaIndex and LangChain Understanding of MLOps or LLMOps. Familiarity with Agile methodologies, preferably Scrum We are actively seeking candidates for full-time, remote work within the UK. #J-18808-Ljbffr More ❯

Experience of quantitative or automated systems development e.g. in a hedge fund or investment bank Expertise in building distributed systems with large data warehouses and both on-line and batch processing Experience of web-based development and visualisation technology for portraying large and complex data sets and relationships Substantial quant development engineering experience with relevant mathematical knowledge e.g. More ❯

Experience of quantitative or automated systems development e.g. in a hedge fund or investment bank Expertise in building distributed systems with large data warehouses and both on-line and batch processing Experience of web-based development and visualisation technology for portraying large and complex data sets and relationships Substantial quant development engineering experience with relevant mathematical knowledge e.g. More ❯

Responsibilities: Design, implement, and maintain robust ETL pipelines to integrate data from diverse sources, including APIs like Facebook, Google Analytics, and payment providers. Develop and optimize data models for batch processing and real-time streaming using tools like AWS Redshift, S3, and Kafka. Lead efforts in acquiring, storing, processing, and provisioning data to meet evolving business requirements. More ❯

company. Responsibilities Partner with stakeholders across Product, Data, and Engineering to translate strategic goals into platform capabilities. Lead and mentor 1-2 squads responsible for experimentation, analytics event logging, batch data platform, real-time infrastructure, data observability and governance. Collaborate closely with stakeholders to define and drive the technical roadmap for Depop's modern data platform, enabling reliable and … Deep understanding of distributed systems and modern data ecosystems - including experience with Databrick, Apache Spark, Apache Kafka and DBT. Demonstrated success in managing data platforms at scale, including both batch processing and real-time streaming architectures. Deep understanding of data warehousing concepts, ETL/ELT processes, and analytics engineering. Strong programming skills, particularly in Python, Scala or Java More ❯

The Data Platform team at Pismo is responsible for integrating data produced by Pismo's Banking as a Service/Payments platform with its clients, whether through batch processing (files) or near real-time (events). With a robust, low-latency, and 100% cloud-based architecture, the team excels in technology and fosters a strong culture of collaboration … on-call rotations, promptly addressing and resolving production incidents, while proactively implementing preventative measures. Build and optimize ETL/ELT processes to handle large-scale data, leveraging advanced data processing and distributed systems techniques. Maintain rigorous data governance and platform standards, working closely with data stakeholders to ensure consistent, high-quality data assets. Introduce and adopt new technologies, tools … improve performance, and reduce costs. Minimum Qualifications Technical Skills: 4+ Years experience as a Data Engineer Strong experience of AWS Services (eg Lambda, Kinesis, SQS, Firehose, and S3) Stream Processing and Delivery: Kinesis, Confluent Cloud, Amazon SNS, SQS and Event Bridge. Monitoring Tools: Grafana and Cloudwatch. (Logs/tracing/spans/monitor/alerts/dashboards) SQL experience More ❯

and performant data engineering solutions, working closely with squad members and stakeholders. Handle data in multiple formats (structured, semi-structured, and unstructured) and manage data at various latencies - from batch processing to real-time streaming. Collaborate cross-functionally across the business to enable self-service analytics and innovation through well-organised and accessible data assets. Ensure engineering solutions … understanding of cloud-based data architectures (e.g., data lakes, data vaults, data warehouses). Experience in working with Agile frameworks such as Scrum, Kanban, Lean. Experience with distributed data processing platforms such as Databricks. Understanding of data modelling and architectures (e.g. data vaults, data warehousing, data lakes etc.) Version control and best practices using Github. Strong communication and collaboration More ❯

they operate. Hands on Experience in Java , Spark , Scala ( or Java) Production scale hands-on Experience to write Data pipelines using Spark/any other distributed real time/batch processing. Strong skill set in SQL/Databases Strong understanding of Messaging tech like Kafka, Solace , MQ etc. Writing production scale applications to use the Caching technologies. Understanding of More ❯

they operate. Hands on Experience in Java , Spark , Scala ( or Java) Production scale hands-on Experience to write Data pipelines using Spark/any other distributed real time/batch processing. Strong skill set in SQL/Databases Strong understanding of Messaging tech like Kafka, Solace , MQ etc. Writing production scale applications to use the Caching technologies. Understanding of More ❯

Responsibilities: Design, implement, and maintain robust ETL pipelines to integrate data from diverse sources, including APIs like Facebook, Google Analytics, and payment providers. Develop and optimize data models for batch processing and real-time streaming using tools like AWS Redshift, S3, and Kafka. Lead efforts in acquiring, storing, processing, and provisioning data to meet evolving business requirements. More ❯

Responsibilities: Design, implement, and maintain robust ETL pipelines to integrate data from diverse sources, including APIs like Facebook, Google Analytics, and payment providers. Develop and optimize data models for batch processing and real-time streaming using tools like AWS Redshift, S3, and Kafka. Lead efforts in acquiring, storing, processing, and provisioning data to meet evolving business requirements. More ❯

Responsibilities: Design, implement, and maintain robust ETL pipelines to integrate data from diverse sources, including APIs like Facebook, Google Analytics, and payment providers. Develop and optimize data models for batch processing and real-time streaming using tools like AWS Redshift, S3, and Kafka. Lead efforts in acquiring, storing, processing, and provisioning data to meet evolving business requirements. More ❯

tools in a fast-moving, high-performance environment. Your responsibilities may include: Data Pipeline Development Build and maintain scalable, efficient ETL/ELT pipelines for both real-time and batch processing. Integrate data from APIs, streaming platforms, and legacy systems, with a focus on data quality and reliability. Infrastructure & Architecture Design and manage data storage solutions, including databases, warehouses … and lakes. Leverage cloud-native services and distributed processing tools (e.g., Apache Flink, AWS Batch) to support large-scale data workloads. Operations & Tooling Monitor, troubleshoot, and optimize data pipelines to ensure performance and cost efficiency. Implement data governance, access controls, and security measures in line with best practices and regulatory standards. Develop observability and anomaly detection tools to … maintaining scalable ETL/ELT pipelines and data architectures. Hands-on expertise with cloud platforms (e.g., AWS) and cloud-native data services. Comfortable with big data tools and distributed processing frameworks such as Apache Flink or AWS Batch. Strong understanding of data governance, security, and best practices for data quality. Effective communicator with the ability to work across technical More ❯

scale data migration and modernization initiatives. Architect end-to-end data platforms using Dataproc, Dataflow, Pub/Sub, BigQuery, Cloud Spanner, and Bigtable. Define and implement real-time and batch processing pipelines for complex use cases involving streaming analytics, ML feature engineering, and automation. Act as a trusted advisor to senior technical and business stakeholders across industries (e.g. … BigQuery, Cloud Spanner, and Bigtable. Hands-on experience with orchestration tools like Apache Airflow or Cloud Composer. Hands-on experience with one or more of the following GCP data processing services: Dataflow (Apache Beam), Dataproc (Apache Spark/Hadoop), or Composer (Apache Airflow). Proficiency in at least one scripting/programming language (e.g., Python, Java, Scala) for data More ❯

scale data migration and modernization initiatives. Architect end-to-end data platforms using Dataproc, Dataflow, Pub/Sub, BigQuery, Cloud Spanner, and Bigtable. Define and implement real-time and batch processing pipelines for complex use cases involving streaming analytics, ML feature engineering, and automation. Act as a trusted advisor to senior technical and business stakeholders across industries (e.g. … BigQuery, Cloud Spanner, and Bigtable. Hands-on experience with orchestration tools like Apache Airflow or Cloud Composer. Hands-on experience with one or more of the following GCP data processing services: Dataflow (Apache Beam), Dataproc (Apache Spark/Hadoop), or Composer (Apache Airflow). Proficiency in at least one scripting/programming language (e.g., Python, Java, Scala) for data More ❯

scale data migration and modernization initiatives. Architect end-to-end data platforms using Dataproc, Dataflow, Pub/Sub, BigQuery, Cloud Spanner, and Bigtable. Define and implement real-time and batch processing pipelines for complex use cases involving streaming analytics, ML feature engineering, and automation. Act as a trusted advisor to senior technical and business stakeholders across industries (e.g. … BigQuery, Cloud Spanner, and Bigtable. Hands-on experience with orchestration tools like Apache Airflow or Cloud Composer. Hands-on experience with one or more of the following GCP data processing services: Dataflow (Apache Beam), Dataproc (Apache Spark/Hadoop), or Composer (Apache Airflow). Proficiency in at least one scripting/programming language (e.g., Python, Java, Scala) for data More ❯

scale data migration and modernization initiatives. Architect end-to-end data platforms using Dataproc, Dataflow, Pub/Sub, BigQuery, Cloud Spanner, and Bigtable. Define and implement real-time and batch processing pipelines for complex use cases involving streaming analytics, ML feature engineering, and automation. Act as a trusted advisor to senior technical and business stakeholders across industries (e.g. … BigQuery, Cloud Spanner, and Bigtable. Hands-on experience with orchestration tools like Apache Airflow or Cloud Composer. Hands-on experience with one or more of the following GCP data processing services: Dataflow (Apache Beam), Dataproc (Apache Spark/Hadoop), or Composer (Apache Airflow). Proficiency in at least one scripting/programming language (e.g., Python, Java, Scala) for data More ❯

scale data migration and modernization initiatives. Architect end-to-end data platforms using Dataproc, Dataflow, Pub/Sub, BigQuery, Cloud Spanner, and Bigtable. Define and implement real-time and batch processing pipelines for complex use cases involving streaming analytics, ML feature engineering, and automation. Act as a trusted advisor to senior technical and business stakeholders across industries (e.g. … BigQuery, Cloud Spanner, and Bigtable. Hands-on experience with orchestration tools like Apache Airflow or Cloud Composer. Hands-on experience with one or more of the following GCP data processing services: Dataflow (Apache Beam), Dataproc (Apache Spark/Hadoop), or Composer (Apache Airflow). Proficiency in at least one scripting/programming language (e.g., Python, Java, Scala) for data More ❯

on Azure (e.g., Azure Data Lake, Synapse, Data Factory) and AWS (e.g., Redshift, Glue, S3, Lambda) . Strong background in data architecture , including data modeling, warehousing, real-time and batch processing, and big data frameworks. Proficiency with modern data tools and technologies such as Spark, Databricks, Kafka, or Snowflake (bonus). Knowledge of cloud security, networking, and cost More ❯

on Azure (e.g., Azure Data Lake, Synapse, Data Factory) and AWS (e.g., Redshift, Glue, S3, Lambda) . Strong background in data architecture , including data modeling, warehousing, real-time and batch processing, and big data frameworks. Proficiency with modern data tools and technologies such as Spark, Databricks, Kafka, or Snowflake (bonus). Knowledge of cloud security, networking, and cost More ❯

on Azure (e.g., Azure Data Lake, Synapse, Data Factory) and AWS (e.g., Redshift, Glue, S3, Lambda) . Strong background in data architecture , including data modeling, warehousing, real-time and batch processing, and big data frameworks. Proficiency with modern data tools and technologies such as Spark, Databricks, Kafka, or Snowflake (bonus). Knowledge of cloud security, networking, and cost More ❯

on Azure (e.g., Azure Data Lake, Synapse, Data Factory) and AWS (e.g., Redshift, Glue, S3, Lambda) . Strong background in data architecture , including data modeling, warehousing, real-time and batch processing, and big data frameworks. Proficiency with modern data tools and technologies such as Spark, Databricks, Kafka, or Snowflake (bonus). Knowledge of cloud security, networking, and cost More ❯

client is a Series C Silicon Valley HealthTech with teams in San Francisco and London. We are hiring a Backend Engineer to architect and build scalable systems for ingesting, processing, and managing diverse health data sources. You will work on the backbone of a platform that combines real-time data, AI insights, and personalized care pathways to transform healthcare … delivery. Core Responsibilities: Design and implement scalable backend services that ingest, transform, and persist diverse health data Build APIs and data pipelines for real-time and batch processing, ensuring data integrity, security, and compliance . Integrate with external systems and protocols including FHIR APIs, BLE-based devices, Apple HealthKit, and Google Fit . Ensure backend services are designed More ❯

engineering, and stakeholder management in investment banking environment. Key Responsibilities Design and maintain Power BI dashboards for trading, risk, and regulatory reporting Build data pipelines for real-time and batch processing of financial data Partner with traders, portfolio managers, and risk teams to deliver analytics solutions Ensure compliance with regulatory reporting requirements Optimize data models for front office More ❯