14 of 14 Apache Jobs in the City of London

making AI tools a core part of how you work. Experience with any part of our data stack would be beneficial: Django REST Framework, Apache Airflow, Apache Spark, Apache Iceberg, Apache Kafka, Terraform, and AWS (EMR, ECS). We manage much of this ourselves, so experience ...

documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building and maintaining ...

years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

support Workflow orchestration (Airflow or Dagster) AWS data services (Athena, S3, MWAA, Iceberg) Data quality monitoring & incident management Nice to Have: Airbyte DBT Apache Spark Grafana FTP/data distribution platforms Financial data experience ...

will have strong experience in data engineering, with advanced proficiency in SQL and Python, alongside exposure to large-scale data processing frameworks such as Apache Spark. You should have a solid understanding of data architecture, data wrangling and data quality practices, as well as experience working within financial markets ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement Why Apply ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Work with solution architects to design migration-day data processes for partner onboarding. Build and maintain ETL pipelines and workflow automation using Snowflake and Apache Airflow. Document processes, learnings, and development work using version control best practices. Implement robust data quality checks and reconciliation processes throughout pipelines. Collaborate with … functionally. Strong communication skills, with the ability to translate technical concepts to non-technical audiences. Hands-on experience building and managing production workflows in Apache Airflow. Experience delivering production-grade transformations in dbt, including testing and documentation. Advanced SQL skills across multiple dialects. Strong understanding of data modelling ...

analysis on complex system-wide issues. Linux Engineering: Manage and optimise a large-scale estate across RedHat, Ubuntu, and CentOS, ensuring peak performance of Apache, Nginx, and Tomcat stacks. Automation & IaC: Drive efficiency through the development of custom Ansible, Puppet, and Terraform scripts to automate deployments and server hardening. ...

orchestrate workloads across CPU and GPU environments. Your work will include: Designing and building distributed services in Rust Developing scalable work queue systems (e.g. Apache Pulsar, RabbitMQ) Building GPU node orchestration services for data movement Creating robust storage lifecycle and indexing services Designing performant APIs using gRPC What ...

services including Vertex AI, Model Garden, BigQuery, Cloud Run and Google’s ADK. • Experience with document and data processing using tools such as Apache Tika, plus storage or caching technologies like Postgres, Redis, GCS or BigQuery. • Solid DevOps practices including CI/CD, Docker, observability and automated testing. ...

site - £520-550pd via Umbrella Contract until Feb 27 Key Responsibilities ETL Pipeline Architecture : Design, build, and maintain scalable ETL pipelines using Apache Airflow, Python, PySpark, and AWS Glue to transform complex financial datasets into actionable insights. Data Integration : Lead the integration of diverse data sources, ensuring data … Data Stack : Deep hands-on experience with AWS Glue and related AWS services (S3, Lambda, Athena, Glue Catalog). Orchestration: Proven experience with Apache Airflow for scheduling, monitoring, and managing complex workflows. Docker: Proficiency in containerization using Docker and experience with AWS container services. Nice-to-Have Skills Capital ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark. Very useful - Python, SQL, JSON, LakeFS. Reasonable Adjustments ...