1 to 25 of 302 Apache Jobs

services for personalisation· Developing Java/Spring Boot microservices with REST/gRPC APIs· Working with Kafka and event-driven architectures· Building pipelines using Apache Flink and Apache Beam· Integrating ML models and feature stores· Working with NoSQL databases and Redis caching· Using Apache Camel for integration … Familiarity with stream processing concepts· Knowledge of distributed systems· Strong communication and teamwork skills· Interest in data-driven personalisation systemsNice to have· Experience with Apache Flink and Apache Beam· Experience with Kafka· Familiarity with Apache Camel· Experience with NoSQL databases (Cassandra, DynamoDB) and Redis· Cloud experience ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing ...

proficiency in SQL and database technologies Experience with programming languages such as Python, Java, or Scala Hands-on experience with data pipeline tools (e.g., Apache Airflow, Kafka) Familiarity with cloud platforms (AWS, Azure, Google Cloud) Understanding of data warehousing solutions (e.g., Snowflake, Redshift, BigQuery) Strong problem-solving and analytical ...

supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting ...

supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

analysis and returning the results to various business systems.We build on a vast variety of technological stacks, from building products on top of Apache Kafka, Apache Spark, and Apache Beam to integrating Neo4j into Google Dataflow Templates and Microsoft Fabric. We also build backend components … ideally have professional experience developing software systems, either in cloud or on-premises, involving either of the following;Distributed event streaming platforms, such as Apache Kafka or similar products,Data processing and analytics platforms, such as Apache Spark, Apache Beam, or their derivatives,Data warehouses ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

React, Node) using Webpack, Typescript, and GitHub.Experience with Docker and/or Kubernetes and containerized deployments (AKS).Experience with workflow orchestration tools such as Apache Airflow or n8n.Experience of code quality, test, CI/CD and package managers such as Jest, Lint, Brigade, and Helm.Experience with Kafka and event ...

using Webpack, Typescript, and GitHub. Experience with Docker and/or Kubernetes and containerised deployments (AKS). Experience with workflow orchestration tools such as Apache Airflow or n8n. Experience of code quality, test, CI/CD and package managers such as Jest, Lint, Brigade, and Helm. Experience with Kafka ...

clean, maintainable, and efficient code, ensuring adherence to best practices through code reviewsBig Data EngineeringDesign, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache AirflowOptimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performanceDevelop and fine-tune complex queries using Apache … related field (or equivalent experience)8+ years of experience in software and/or data engineering with expertise in big data technologies such as Apache Spark and Apache AirflowAdvanced SQL skills, with expertise in query optimisation for large datasetsExpertise in developing APIs and robust backend services, with strong ...

efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries using … Apache Spark for large-scale datasets Monitor, troubleshoot, and improve data systems to minimise downtime and maximise efficiency Collaboration & Mentorship Partner with analytics, software engineering, and other teams to deliver integrated, high-quality solutions Provide technical guidance and mentorship to junior engineers, promoting best practices in software and data ...

expected to be up to 10% but may increase over time as the business evolves.Desired QualificationsExperience working with distributed data processing frameworks, including Apache Spark.Exposure to advanced analytics or AI‐adjacent data use cases, including preparing data for machine learning or intelligent applications.Familiarity with additional Azure services such ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding experience, such ...

Advanced experience operating Delta tables at scale, covering transaction semantics, schema evolution, partitioning, and compaction, alongside highly optimised analytical query execution using DuckDB or Apache Spark.• Production-grade relational database expertise, particularly PostgreSQL (query tuning, indexing, partitioning, replication, and schema design across OLTP and analytics).• Strong KDB experience ...

measurable value.• Excellent analytical, problem-solving, and communication skills.• Proven experience leading data modernisation initiatives across multiple sectors.Technical Expertise• Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog.• Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation).• Hands-on expertise ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal experience ...

testing, and deployment.Deep understanding of Quantexa components such as:Entity Resolution (ER)Scoring FrameworkContextual Network GenerationUI ConfigurationStrong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearchExperience with data modelling and design in the context of large-scale analytics platforms.Strong stakeholder management and communication skills – capable of interfacing ...