1 to 25 of 189 Apache Jobs in London

computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers operate … want to advance the capabilities of modern data systems and apply that work to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog interoperability ...

computations are represented, how query plans are optimised, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers operate … learn how modern data systems work from the inside and apply that knowledge to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog interoperability ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing ...

experience as a Data Engineer or Senior Data Engineer Expert-level Python development skills Extensive hands-on experience with Databricks in enterprise environments Strong Apache Spark and PySpark expertise Advanced SQL development and optimisation skills Experience designing and building large-scale ETL and data integration solutions Strong understanding ...

Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such as Jenkins ...

software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best practices ...

analysis and returning the results to various business systems. We build on a vast variety of technological stacks, from building products on top of Apache Kafka, Apache Spark, and Apache Beam to integrating Neo4j into Google Dataflow Templates and Microsoft Fabric. We also build backend components … ideally have professional experience developing software systems, either in cloud or on-premises, involving either of the following; Distributed event streaming platforms, such as Apache Kafka or similar products, Data processing and analytics platforms, such as Apache Spark, Apache Beam, or their derivatives, Data warehouses ...

pipelines, deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. ...

pipelines, deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding experience ...

production pipelines Strong SQL and Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. APIs and cloud business system integration. Data governance, lineage and quality frameworks. What We Offer A hands-on role developing ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like JSON ...

SQL. Well-versed in software engineering and systems design principles. Hands-on experience with Azure cloud data platforms is essential. Familiarity with Spark and Apache Airflow is required. Proficient in Linux/Unix/*nix environments. Strong knowledge of ETL tools and data modelling techniques. Candidates will need ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create data ...

gateways for AI applications Infrastructure as Code experience with Terraform, Ansible, or other IaC tools Experience building and managing data pipelines using tools like Apache Spark AI safety and governance experience with model evaluation, LLM judges, bias detection, and responsible AI practices Familiarity with statistical fundamentals and running ...

Hands‐on experience as a Data Engineer or Data Platform Developer, including direct contributions to production codebases. Expert‐level proficiency in Python, SQL, and Apache Spark for data pipeline development. Proven experience building ELT workflows using Fivetran, Airflow, DBT, or Databricks. Solid understanding of data modeling, dimensional design ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with ...

similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team Comfortable in a fast-paced environment Have excellent written and verbal English skills Last ...

designs, explaining technical trade‐offs and reviewing others’ work The knowledge, experience and qualifications that will help Ingestion and orchestration tools such as Ansible, Apache NiFi or Cribl Docker, Kubernetes and AWS services such as S3, Lambda, Step Functions and SageMaker Working with Cyber Security and Infrastructure data, such ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...