1 to 25 of 38 Permanent Apache Iceberg Jobs in England

massive scale? Amazon S3 is the storage for the internet, and Amazon S3 Tables is the first cloud object store with built-in Apache Iceberg support, purpose-built for storing and querying tabular data at massive scale. It is fully managed, continuously optimizing, and delivers … higher transactions per second than Iceberg tables on general purpose S3 buckets, all within the same foundation that gives S3 eleven nines of durability. We are on the lookout for the curious, those who think big and want to define the world of tomorrow. Customers are bringing their most ...

management, and quality assurance.Collaboration: Engage with business and technology teams to align the platform with organizational goals.Mandatory Skillsets: Python, Snowflake, dbt, dremio, glue jobs, Apache iceberg, AWS ,SQL.Preferred Technical Skills:Programming: 5+ years of experience in Python/Java.Cloud Expertise: Strong understanding of AWS services (e.g., Lambda, Step … Functions, ECS).Data Platforms: Hands-on experience with Snowflake and data stack technologies like Apache Iceberg and Spark.Workflow Orchestration: Exposure to tools like Apache Airflow, Prefect, Dagster, or DBT.Data Services: Familiarity with AWS Glue, Lake Formation, EMR, EventBridge, Athena, and similar services.Metadata Tools: Experience with tools like ...

practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing migration solutions, leveraging technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML-based configurations. You will thrive in a collaborative, client-facing environment, with a passion for solving … target cloud architecturesData Pipeline Testing & Validation (Core Focus)Test end-to-end ETL data solutions running on AWS services, including AWS Glue and Apache IcebergValidate data feed pipeline migrations transitioning to new AWS Data Lakehouse architecturesTest pipelines underpinned by: AI ETL acceleratorsPython/PySpark transformationsSQL-based logicYAML configuration-driven ...

semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. AI-Accelerated Schema & Pipeline … Change Data Capture (CDC), clustering, data migration, and aggregation. Use AI query-optimization tools to analyse execution plans, auto-tune complex Snowflake/Iceberg SQL workloads, and eliminate performance bottlenecks. Intelligent Telemetry & Closed-Loop Reasoning: Integrate automated AI inferencing and reasoning layers directly into data pipelines to detect telemetry ...

activities. You will work closely with architects, senior engineers, and analysts to deliver scalable and reliable migration solutions, using technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML configurations. You should be comfortable working in a collaborative, delivery-focused environment and have a strong … repointing and migration to cloud environmentsData Pipeline Testing & Validation (Core Focus)Test ETL/ELT data pipelines on AWS services, including AWS Glue and Apache IcebergSupport validation of data pipeline migrations to AWS Data Lakehouse architecturesTest pipelines using: Python/PySpark transformationsSQL-based validation logicYAML-driven configurationsExecute and validate ...

platforms capable of processing billions of events per day and managing terabytes scale datasets. Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino. Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment … OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code. Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability. Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure ...

operating ETL/ELT pipelines and large‐scale data architectures. Hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

environments. Technical Skills: Experience building and operating ETL/ELT pipelines. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

tooling, or cloud cost management across at least one major cloud provider (Azure, AWS, or GCP). Technical Skills Data engineering: strong proficiency with Apache Spark, dbt, Airflow/Azure Data Factory or equivalent orchestration, Data Lake/Apache Iceberg, and advanced SQL for large‐scale transformations. ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root-cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark. Working knowledge of common data formats such as JSON, Avro and Parquet. For More Experienced CandidatesStronger ownership of technical design across multiple datasets … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, KafkaData formats: JSON, Avro, ParquetPlatforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQEngineering and deployment: CI/ ...

skills Hands-on experience with Databricks for scalable data engineering, analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimization Experience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure … data pipeline deployment and monitoring using CI/CD tools and infrastructure-as-code practices Hands-on expertise with distributed data processing frameworks (e.g., Apache Spark, Hadoop) and real-time data streaming technologies (e.g., Kafka, Kinesis, Spark) Track record of collaborating with cross-functional teams (engineering, analytics, business stakeholders ...

Medallion and Data Products. Hands on mindset : Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON. Agile Ways of Working Run and contribute to agile ceremonies, helping to shape and improve the agile process Work with the project ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with emerging technologies in data ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with AI evaluation and observability ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you'll learn ...

regulated financial organisation. Cloud experience on AWS and/or GCP, with AWS preferred. Experience with tools such as S3, Lambda, Glue, Athena, Iceberg, Vertex AI, BigQuery, GCS, Cloud Run, or Pub/Sub. Experience with AI/ML observability and evaluation tools such as MLflow, LangSmith ...

index consumption experience. Develop efficient, and secured cloud-native architectures. Work with relational databases (PostgresSQL), possibly NoSQL and object stores, and modern Lakehouse architectures (Iceberg)Design scalable data models, jointly with data governanceContribute to code reviews, mentor peers and more junior members of the team, and help shape ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt)Experience designing and operating orchestration pipelines using Airflow ...

pipelines, inference serving, and GenAI/RAG architectures grounded in proprietary content. Lakehouse and modern data platform design: strong knowledge of open table formats (Apache Iceberg, Delta Lake), zone architecture (landing, cleansed, curated), streaming and batch pipeline patterns, and data quality frameworks. Project engagement and design assurance: able ...

index consumption experience. Develop efficient, and secured cloud-native architectures. Work with relational databases (PostgresSQL), possibly NoSQL and object stores, and modern Lakehouse architectures (Iceberg)Design scalable data models, helped by peers and data governanceContribute to code reviews and be code reviewed to learn and improvePartner with global teams ...

latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information ...

Hadoop ecosystem and deep understanding of components like Spark, Flink, Elasticsearch, Doris, ClickHouse, and Trino., Strong knowledge of big data architecture, including Data Lake (Iceberg, Hudi) and Serverless data concepts. Familiarity with mainstream global cloud products such as AWS EMR/Redshift, Google BigQuery, Snowflake, and Databricks., Ability ...

implementing computational improvements Making improvements and/or fixing bugs by submitting patches to open-source projects in our stack, such as Spark, Iceberg, Hadoop, Airflow, and many others Acting as an expert on Spark and Scala Mentoring engineers (including through pairing, design discussions, and knowledge-sharing sessions ...