9 of 9 Permanent Apache Jobs in West London

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

with distributed event stores and stream-processing platforms such as Kafka of Kinesis. Experience working with batch processing frameworks, such as DBT, Argo Workflows, Apache Airflow, etc. Familiarity with Docker, Kubernetes, Amazon EKS Familiarity with Continuous Integration with GitHub Actions Familiarity with Test-Driven Development and XP Our commitment ...

Experience working with distributed event stores and stream-processing platformssuch as Kafka of Kinesis.● Experience working with batch processing frameworks, such as DBT, ArgoWorkflows, Apache Airflow, etc.● Familiarity with Docker, Kubernetes, Amazon EKS● Familiarity with Continuous Integration with GitHub Actions● Familiarity with Test-Driven Development and XPOur commitment ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data Meshes ...

data lake houses and data mesh Strong understanding of best practice DataOps and MLOps Up-to-date understanding of various data engineering technologies including Apache Spark, Databricks and Hadoop Strong understanding of agile ways of working Up-to-date understanding of various programming languages including Python, Scala ...

Build and operate batch and streaming data pipelines with strong scalability, performance, and fault tolerance Develop and manage workflow orchestration using tools such as Apache Airflow to support reliable, observable, and well-scheduled data movement and transformations Implement and optimize data models and warehouse structures to support analytics … Beam, or similar) Experience automating deployment, releases, and testing in continuous integration and continuous delivery pipelines Experience with lakehouse patterns and table formats (e.g., Apache Iceberg) Experience with federated query engines such as Trino Experience designing automated tests (unit, component, integration, and end-to-end), including use of mocking ...

databases Experience with one of Azure/AWS/GCP ETL tools like Informatica, Alteryx and Business Glossary tools like Collibra, Azure Purview or Apache Atlas. TDD/BDD, Experience working in an Agile development environment Identity management and synchronisation will be an add on for the resources Experience ...

with cloud data warehouses such as Snowflake or Databricks Experience in building ML features Any experience in real time streaming tools and frameworks, e.g. Apache Flink, is a plus Any experience evaluating new tools and processes to drive the adoption and usability of AI and ML ready data Familiarity ...