11 of 11 Apache Jobs in the City of London

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit your ...

documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building and maintaining ...

support Workflow orchestration (Airflow or Dagster) AWS data services (Athena, S3, MWAA, Iceberg) Data quality monitoring & incident management Nice to Have: Airbyte DBT Apache Spark Grafana FTP/data distribution platforms Financial data experience ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design and delivery ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … monitoring. Required Skills 8–10 years ’ experience in data engineering. Strong hands-on experience with: AWS S3, Athena, Glue, Redshift, EMR/Spark Apache Iceberg Python/SQL Experience in telecom data pipelines and handling large-scale structured/semi-structured data. Strong problem-solving, optimization, and debugging skills. ...

will have strong experience in data engineering, with advanced proficiency in SQL and Python, alongside exposure to large-scale data processing frameworks such as Apache Spark. You should have a solid understanding of data architecture, data wrangling and data quality practices, as well as experience working within financial markets ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Work with solution architects to design migration-day data processes for partner onboarding. Build and maintain ETL pipelines and workflow automation using Snowflake and Apache Airflow. Document processes, learnings, and development work using version control best practices. Implement robust data quality checks and reconciliation processes throughout pipelines. Collaborate with … functionally. Strong communication skills, with the ability to translate technical concepts to non-technical audiences. Hands-on experience building and managing production workflows in Apache Airflow. Experience delivering production-grade transformations in dbt, including testing and documentation. Advanced SQL skills across multiple dialects. Strong understanding of data modelling ...

analysis on complex system-wide issues. Linux Engineering: Manage and optimise a large-scale estate across RedHat, Ubuntu, and CentOS, ensuring peak performance of Apache, Nginx, and Tomcat stacks. Automation & IaC: Drive efficiency through the development of custom Ansible, Puppet, and Terraform scripts to automate deployments and server hardening. ...

design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process Collaborate … heavy data role Excellent communication skills, able to translate technical knowledge to non-technical colleagues Hands-on experience building and operating production workflows in Apache Airflow Hands-on experience delivering production-grade transformations in dbt, including tests and documentation Strong SQL skills and experience working across multiple SQL dialects. ...

services including Vertex AI, Model Garden, BigQuery, Cloud Run and Google’s ADK. • Experience with document and data processing using tools such as Apache Tika, plus storage or caching technologies like Postgres, Redis, GCS or BigQuery. • Solid DevOps practices including CI/CD, Docker, observability and automated testing. ...