1 to 25 of 96 Apache Jobs

Python, Java or Scala, and SQL. Hands-on experience building and operating large-scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. Experience working with cloud-native data platforms ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

data quality and consistency across platforms while complying with governance policies.What You Bring:Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs.Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB, Cosmos ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools (e.g. ...

design approaches.You understand where traditional patterns remain appropriate – and where modern approaches add measurable value.Data Engineering & Platform Understanding:Strong knowledge of distributed processing (e.g. Apache Spark).Familiarity with Hadoop ecosystem components.Working knowledge of SQL and NoSQL technologies.Understanding of ETL/ELT design trade-offs.Awareness of DevOps practices ...

Computer Science, Engineering, Information Technology or equivalent work experience. • Strong knowledge and hands on experience of big data technologies such as data lakes, Apache Spark, Flink, Kafka etc. • Good knowledge of data warehousing concepts and ETL processes, DevOps, CI/CD and software testing. • Expertise with one of programming ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

/CD, DevOps, and Infrastructure as Code practices for data platforms. Metadata management, lineage tooling, and data observability/monitoring experience. Familiarity with dbt, Apache Airflow, Azure Data Factory, Kafka, Event Hubs, or comparable orchestration/integration tools. Familiarity with enterprise AI platforms — Azure AI, AWS SageMaker, Google Vertex ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security, and lineage best ...

design data architectures that facilitate model training and inference.Data Governance Tools: Familiarity with data governance or data cataloguing tools (such as Collibra, Alation, or Apache Atlas) and lineage-tracking systems. Hands-on experience setting up or maintaining a data catalogue, documenting data definitions, or automating data lineage capture ...

expertise in data pipeline development, ETL/ELT processes, and data warehousing solutions.Proficient with cloud platforms (AWS, GCP, Azure) and big data tools like Apache Spark, Python, etc.Experience with relational and NoSQL databases.Solid understanding of data modeling, data architecture, engineering frameworksProven ability to lead and scale engineering teams ...

using Databricks REST APIs, Prometheus, or Azure Monitor.• Maintain best practices for data engineering performance, testing, and documentation.Technical Skills Required• Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake.• Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies.• Experience tuning job configurations (e.g. ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with emerging technologies in data engineering ...

platform and lakehouse transformation journeys What We're Looking For Strong commercial experience in Data Engineering Hands-on experience with Databricks and/or Apache Spark Strong PySpark and SQL development skills Experience designing, building and supporting data pipelines Experience working with Azure or AWS Good understanding of modern ...

market data engineering with proven experience building reliable, scalable data platforms. Our non-negotiable requirements are: Expert Python Strong SQL Scalable data pipeline development Apache Parquet Market data normalisation Automated data quality controls Deep tick and market data expertise Experience in MLOps is advantageous but not essential. We welcome ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with emerging technologies in data engineering ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance practices ...

SnowflakeDeep hands-on experience with Spark/PySpark and other big data processing technologiesExpertise in open-source table formats and catalog services such as Apache IcebergExperience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI)Familiarity with emerging technologies in data engineeringAbility ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with AI evaluation and observability practices ...

managing data pipelines, ETL processes, and data architecture. Proficiency in programming languages such as SQL and Python, and experience with big data technologies like Apache Spark or Hadoop. Experience with database management systems, both relational and non-relational. Understanding of data modeling and warehousing principles. Ability to collaborate effectively ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

with dbt or Airbyte is preferred. Experience developing production Spark or Hadoop applications Gained experience with a workload scheduling and orchestration tool such as Apache Airflow/Cloud Composer Familiarity with modern DataOps (e.g. quality eng, testing, pipeline deployments etc) practices and IaC (e.g. Terraform) Familiarity with the constellation ...