26 to 50 of 91 Apache Iceberg Jobs

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

seniority. Knowledge and Experience - essential - AWS Glue ETL - AWS Glue Data Quality - AWS Lambda - AWS Lake Formation - Medallion Architecture based ETL processes using S3, Iceberg and Athena - Azure DevOps/AWS CodeCommit/AWS CodePipeline - AWS Step functions - Athena SQL ; Spark SQL ; Python - AWS CLI - Release management (CI/ ...

environment Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Experience working with lakehouse/medallion architectures in Databricks Experience with experimentation support tooling (such as Optimizely) A genuine curiosity about ...

Structured Streaming, and Lakehouse architecture Strong hands-on experience with Snowflake, including data modelling, performance optimisation, Snowpark, data sharing, and governance capabilities Expertise in Apache Spark, Python, SQL, and modern ELT/ETL frameworks Proven ability to rapidly evaluate, prototype, and compare emerging technologies and architectural patterns Experience designing … Experience with AI/ML platforms, feature stores, and MLOps frameworks Experience evaluating alternative analytical databases such as DuckDB, ClickHouse, Redshift, BigQuery, KDB+, or Apache Iceberg-based architectures Experience with data catalogue and governance platforms Exposure to modern visualisation and BI tools such as Power BI, Tableau, Sigma ...

Medallion and Data Products. Hands on mindset : Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON. Agile Ways of Working Run and contribute to agile ceremonies, helping to shape and improve the agile process Work with the project ...

with Kubernetes-native platforms to deploy and operate data workloads across cloud environments is preferred. Knowledge of Golang would be an advantage. Familiarity with Apache Iceberg table format and schema management via catalogs such as AWS Glue, with an appreciation for data quality and consistency is a strong ...

Development: TypeScript, Angular/React, Python, SQL, OpenAPI. Cloud: AWS (Lambda, API Gateway, S3, SQS, SNS, Aurora, and others as required). Databases: PostgreSQL, Apache Iceberg, DynamoDB, VectorDB (e.g. ChromaDB). IaC and CI/CD: Terraform, AWS CLI, GitLab. AI Development: Agentic AI tooling (e.g. GitHub Copilot ...

modelling, medallion architecture) Leading the design of cloud data platforms (AWS) - including services like Redshift, S3, Glue, Athena Evaluating and introducing new technologies (e.g. Iceberg, Delta Lake, observability tools) Driving data governance, standards, and best practice Acting as a trusted advisor to both technical and business stakeholders What ...

Highly Desirable Skills Experience with streaming frameworks such as Flink, Kafka Streams and Kafka Connect Knowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data Catalog Exposure to DataOps practices and collaboration with Data Engineering teams Familiarity with GPU‐accelerated analytics using Spark with GPUs ...

resolving issuesHighly desirable skillsExperience with streaming frameworks such as Flink, Kafka Streams and Kafka ConnectKnowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data CatalogExposure to DataOps practices and collaboration with Data Engineering teamsFamiliarity with GPU-accelerated analytics using Spark with GPUs or RAPIDSProgramming experience with ...

including dimensional modelling and layered architectures (Gold/Silver/Bronze or equivalent). Production experience with GCP (BigQuery, Dataflow, GCS) and AWS (S3, Iceberg, Kinesis, DynamoDB), you're comfortable operating across both clouds. Familiarity with OLAP and OLTP storage systems and an eagerness to build expertise in stream … processing technologies such as Apache Flink, Kafka, or Spark. (Nice to have: ClickHouse.) Familiarity with observability tooling such as Grafana or Datadog, and a good instinct for keeping systems healthy and well-monitored. Active engagement with the evolving AI/ML tooling landscape — you integrate AI-assisted development tools ...

Amazon S3 AWS Lake Formation Amazon Athena Experience building Lakehouse architectures and modern data platforms. Experience working with data formats such as Parquet and Iceberg tables. Strong experience with Terraform for infrastructure automation. #J-18808-Ljbffr ...

Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well as batch and streaming processing Knowledge of data quality, governance, cataloguing and observability tools (e.g. Datadog), with ...

Analytics, with proven experience as a Senior/Tech Lead setting architectural direction for other engineers Deep, hands‐on knowledge of Snowflake, including ideally Iceberg/open table formats and RBAC. Advanced SQL, Python, and enterprise-grade dbt model development. Experience designing end-to-end data pipelines. Combining batch ...

transparency. Skills & Experience Essential Proven in-depth commercial experience in the following key areas: Databricks- Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning. Apache Spark & Scala- High-performance distributed data processing and optimisation. Cloud Data Engineering (AWS) - Cloud-native data platforms and distributed computing. Data Services Architecture & Engineering … knowledge of client core business functions Demonstrated leadership, project management, and development skills Relationship and consensus building skills Preferred Kafka/Event-Driven Architecture Apache Iceberg/Delta Lake Python AI/ML & LLM integration Financial Services/Capital Markets Education: Bachelor's degree/University degree ...

pipelines, inference serving, and GenAI/RAG architectures grounded in proprietary content. Lakehouse and modern data platform design: strong knowledge of open table formats (Apache Iceberg, Delta Lake), zone architecture (landing, cleansed, curated), streaming and batch pipeline patterns, and data quality frameworks. Project engagement and design assurance: able ...

Tool (DBT) with a demonstrated ability indeveloping data models, contracts, tests, validation, and transformations Experienceworking with modern data distributed file formats (i.e., Parquet, Delta,Iceberg, Hudi) Demonstratedexperience with building data ingestion pipelines from REST API data sources. Strongability to produce technical documentation that can be understood by bothtechnical ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake and engines such as Spark, Flink, or Trino and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt)Experience designing and operating orchestration pipelines using Airflow ...

e.g., Java, TypeScript, SQL) Experience building and consuming APIs and event‐driven services in a cloud environment Experience with data processing frameworks such as Apache Spark and working with large structured datasets Working knowledge of LLM‐based application development, including prompting, tool calling, retrieval, and evaluation Experience using approved … agent frameworks (e.g., Google ADK, LangGraph) and agent protocols such as MCP, A2A, or AG-UI Experience with Databricks, MLflow, and Delta Lake or Apache Iceberg Exposure to model serving on Kubernetes and to model monitoring (drift, latency, accuracy) Familiarity with CRM platforms such as Salesforce and their ...

data services (S3, IAM, Glue, Lake Formation, EventBridge, CloudWatch, Lambda) | Intermediate–Advanced || Security scanning tools (Checkov, tfsec) | Intermediate || Data platforms (Snowflake, Databricks, DBT, Apache Iceberg) | Intermediate || Python (test automation, pipeline utilities) | Intermediate || SRE practices (SLIs/SLOs, chaos engineering, performance testing) | Intermediate || Compliance testing (encryption, tagging, data residency ...

experience with modern cloud data platforms, Spark and distributed data processing Experience with data cataloguing and governance tooling, open table formats (e.g. Delta Lake, Iceberg) and streaming architectures Advanced Python and Microsoft SQL Server development Proven experience leading engineering teams or workstreams Strong stakeholder engagement and consulting skills Working ...

Lead Data EngineerManchester | Hybrid | Circa £85K An established organisation is looking for a hands-on Lead Data Engineer to take ownership of its existing data estate and lead the development of a modern, scalable data ...