1 to 25 of 319 Apache Jobs in England

computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers operate … want to advance the capabilities of modern data systems and apply that work to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog interoperability ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing ...

experience as a Data Engineer or Senior Data Engineer Expert-level Python development skills Extensive hands-on experience with Databricks in enterprise environments Strong Apache Spark and PySpark expertise Advanced SQL development and optimisation skills Experience designing and building large-scale ETL and data integration solutions Strong understanding ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence ...

Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such as Jenkins ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands-on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

analysis and returning the results to various business systems. We build on a vast variety of technological stacks, from building products on top of Apache Kafka, Apache Spark, and Apache Beam to integrating Neo4j into Google Dataflow Templates and Microsoft Fabric. We also build backend components … ideally have professional experience developing software systems, either in cloud or on-premises, involving either of the following; Distributed event streaming platforms, such as Apache Kafka or similar products, Data processing and analytics platforms, such as Apache Spark, Apache Beam, or their derivatives, Data warehouses ...

Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience in Data ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We are looking ...

software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best practices ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

clean, maintainable, and efficient code, ensuring adherence to best practices through code reviewsBig Data EngineeringDesign, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache AirflowOptimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performanceDevelop and fine-tune complex queries using Apache … related field (or equivalent experience)8+ years of experience in software and/or data engineering with expertise in big data technologies such as Apache Spark and Apache AirflowAdvanced SQL skills, with expertise in query optimisation for large datasetsExpertise in developing APIs and robust backend services, with strong ...

efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine‐tune complex queries using … Apache Spark for large‐scale datasets Monitor, troubleshoot, and improve data systems to minimise downtime and maximise efficiency Collaboration & Mentorship Partner with analytics, software engineering, and other teams to deliver integrated, high‐quality solutions Provide technical guidance and mentorship to junior engineers, promoting best practices in software and data ...

pipelines, deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. ...

Advanced experience operating Delta tables at scale, covering transaction semantics, schema evolution, partitioning, and compaction, alongside highly optimised analytical query execution using DuckDB or Apache Spark. Production-grade relational database expertise, particularly PostgreSQL (query tuning, indexing, partitioning, replication, and schema design across OLTP and analytics). Strong KDB experience ...

AWS. Hands-on experience with GCP services and data engineering on GCP. experience with business intelligence and analytics tools such as Looker. Familiarity with Apache ecosystem technologies relevant to data processing and analytics. Background in engineering or computer science, or equivalent practical experience in cloud and data engineering. ...

comfortable operating across both clouds. Familiarity with OLAP and OLTP storage systems and an eagerness to build expertise in stream processing technologies such as Apache Flink, Kafka, or Spark. (Nice to have: ClickHouse.) Familiarity with observability tooling such as Grafana or Datadog, and a good instinct for keeping systems ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like JSON ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like JSON ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration and continuous ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands-on expertise ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal experience ...