1 to 25 of 335 Apache Jobs

detect threats, analyze patterns, and respond to incidents in real-time.You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that process … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teamsBuild and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructureDesign event-driven systems leveraging Kafka for high-throughput, low-latency message streamingTake ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs. Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

data quality and consistency across platforms while complying with governance policies.What You Bring:Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs.Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB, Cosmos ...

development and automation capabilities, to improve the value realized by automationRequired qualifications, skills and capabilitiesExperience with building data pipelines on Databricks and/or Apache Spark in production.Strong coding skills in Python (PySpark) and SQL (Scala a plus).Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema ...

technologies as our stack evolves.Nice to HaveExperience with stream processing frameworks such as Flink or Spark Streaming.Experience with task orchestration tools such as Apache Airflow or similar systems.Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT.Experience working with and designing infrastructures for large-scale data processing ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Experience working with and designing infrastructures for large‐scale data processing (Hive, Snowflake, NoSQL databases). Experience with data governance practices ...

expected to be up to 10% but may increase over time as the business evolves.Desired QualificationsExperience working with distributed data processing frameworks, including Apache Spark.Exposure to advanced analytics or AI‐adjacent data use cases, including preparing data for machine learning or intelligent applications.Familiarity with additional Azure services such ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools (e.g. ...

Advanced experience operating Delta tables at scale, covering transaction semantics, schema evolution, partitioning, and compaction, alongside highly optimised analytical query execution using DuckDB or Apache Spark.• Production-grade relational database expertise, particularly PostgreSQL (query tuning, indexing, partitioning, replication, and schema design across OLTP and analytics).• Strong KDB experience ...

testing, and deployment.Deep understanding of Quantexa components such as:Entity Resolution (ER)Scoring FrameworkContextual Network GenerationUI ConfigurationStrong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearchExperience with data modelling and design in the context of large-scale analytics platforms.Strong stakeholder management and communication skills – capable of interfacing ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.Implement continuous integration and continuous deployment ...

understanding of Quantexa components such as: Entity Resolution (ER), Scoring Framework, Contextual Network Generation, UI Configuration. Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch. Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

Computer Science, Engineering, Information Technology or equivalent work experience. • Strong knowledge and hands on experience of big data technologies such as data lakes, Apache Spark, Flink, Kafka etc. • Good knowledge of data warehousing concepts and ETL processes, DevOps, CI/CD and software testing. • Expertise with one of programming ...

design approaches.You understand where traditional patterns remain appropriate – and where modern approaches add measurable value.Data Engineering & Platform Understanding:Strong knowledge of distributed processing (e.g. Apache Spark).Familiarity with Hadoop ecosystem components.Working knowledge of SQL and NoSQL technologies.Understanding of ETL/ELT design trade-offs.Awareness of DevOps practices ...

testable code.Modern Data & Cloud Practices: Experience with modern data engineering tooling and cloud platforms (ideally AWS and Databricks), and familiarity with distributed processing (e.g. Apache Spark). Exposure to infrastructure as code (such as Terraform and Databricks Asset Bundles) is a plus.Engineering Mindset: Understanding of testing, version control ...

large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security, and lineage best ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create data ...

QualificationsExperience in Data Platforms, Data Engineering, Cloud Engineering, or ML Platform Engineering roles, with exposure to Azure Strong hands-on experience with Python and Apache Spark Experience with Azure data platform technologies such as Azure Databricks, ADLS Gen2, and Unity Catalog Working knowledge of security and access management (RBAC ...