1 to 25 of 46 Apache Iceberg Jobs in London

practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing migration solutions, leveraging technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML-based configurations.You will thrive in a collaborative, client-facing environment, with a passion for solving complex … target cloud architecturesData Pipeline Testing & Validation (Core Focus)Test end-to-end ETL data solutions running on AWS services, including AWS Glue and Apache IcebergValidate data feed pipeline migrations transitioning to new AWS Data Lakehouse architecturesTest pipelines underpinned by:AI ETL acceleratorsPython/PySpark transformationsSQL-based logicYAML configuration-driven ...

data engineering stack, advancing a modern, dbt-based ELT standard on Snowflake and Databricks (with streaming and event-driven patterns on Kafka and Apache Iceberg as the open table format) and migrating legacy pipelines onto it.Establish data observability as a first-class platform capability, leveraging modern observability tooling … support both batch and real-time workloads.Proven success building and modernizing data platforms using Snowflake, Databricks, dbt, Kafka, and open lakehouse formats such as Apache Iceberg, with a strong focus on data quality, lineage, observability, and reliability. Familiarity with modern data observability and quality monitoring tooling ...

activities. You will work closely with architects, senior engineers, and analysts to deliver scalable and reliable migration solutions, using technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML configurations. You should be comfortable working in a collaborative, delivery-focused environment and have a strong … migration to cloud environments Data Pipeline Testing & Validation (Core Focus) Test ETL/ELT data pipelines on AWS services, including AWS Glue and Apache IcebergSupport validation of data pipeline migrations to AWS Data Lakehouse architecturesTest pipelines using: Python/PySpark transformationsSQL-based validation logicYAML-driven configurationsExecute and validate: Initial ...

Fluency: A strong conceptual mastery of modern data platforms. You do not code, but you understand the architectural implications of AWS (S3), Databricks, dbt, Apache Iceberg, and legacy mainframe/SAS ecosystems well enough to guide technical decision-making. Stakeholder Influence: Proven ability to manage C-suite … migrate ~11,000 tables, 4,400 SAS scripts, and 500 BTEQ scripts, ensuring full alignment with the programme’s «Lakehouse» architecture (AWS, Databricks, Apache Iceberg). Strategic Stakeholder Management: Act as the primary liaison between GlobalLogic’s delivery units and senior Barclays stakeholders. Provide transparent reporting on programme ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline lifecycle … development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

building, and operating ETL/ELT pipelines and large-scale data architectures.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential).Ability ...

high-growth environments.Technical Skills:Experience building and operating ETL/ELT pipelines.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

support the tooling to create pipelines that are used for real time decision making in the milliseconds range. Some of the technologies we use:\* Apache Flink\* Apache Iceberg\* K8s operators\* Kafka StreamsWhat do you need We are fully aware that it is uncommon for a candidate ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

capabilities, and skillsHands-on experience with Databricks for scalable data engineering, analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimizationExperience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure … automate data pipeline deployment and monitoring using CI/CD tools and infrastructure-as-code practicesHands-on expertise with distributed data processing frameworks (e.g., Apache Spark, Hadoop) and real-time data streaming technologies (e.g., Kafka, Kinesis, Spark)Track record of collaborating with cross-functional teams (engineering, analytics, business stakeholders ...

reconciliation and root-cause analysis.Experience building or supporting production data pipelines in a collaborative engineering environment.Experience working with distributed data processing frameworks such as Apache Spark.Working knowledge of common data formats such as JSON, Avro and Parquet.Technology EnvironmentThe role will involve working with a modern and evolving data stack. … should bring relevant experience and the ability to work across comparable technologies.Examples of technologies in scope include:Data processing and logic: ANSI SQL, Apache Spark, KafkaData formats: JSON, Avro, ParquetPlatforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQEngineering and deployment: CI/CD tooling ...

this role, you will be the driving force behind building, scaling, and optimizing the critical data pipelines on top of our Lakehouse (Trino + Iceberg) infrastructure following a strict Medallion Architecture (Bronze, Silver, Gold layers). You will build robust data pipelines that serve as the foundational bedrock … working on Technical Leadership & Pipeline EngineeringLakehouse Architecture: Design, build, and optimize robust end-to-end data pipelines on top of our Trino + Iceberg Lakehouse stack.Medallion Implementation: Own the extraction, cleaning, and modeling layers to progressively refine raw event logs (Bronze) into verified subledgers (Silver) and structured Gold metrics.Stream ...

this role, you will be the driving force behind building, scaling, and optimizing the critical data pipelines on top of our Lakehouse (Trino + Iceberg) infrastructure following a strict Medallion Architecture (Bronze, Silver, Gold layers). You will build robust data pipelines that serve as the foundational bedrock … working on? Technical Leadership & Pipeline Engineering Lakehouse Architecture: Design, build, and optimize robust end‐to‐end data pipelines on top of our Trino + Iceberg Lakehouse stack. Medallion Implementation: Own the extraction, cleaning, and modeling layers to progressively refine raw event logs (Bronze) into verified subledgers (Silver) and structured ...

properly — this is the role for you. WHAT YOU'LL BE DOING:Build & deliver Design and develop scalable, testable data pipelines using Python and Apache Spark Build and orchestrate data workflows using AWS services (e.g. Glue, EMR Serverless, Lambda, S3) Contribute to the evolution of a modern lakehouse architecture … e.g. Apache Iceberg) Work with relational databases, particularly PostgreSQL/Aurora PostgreSQL Drive engineering excellence Apply and champion strong software engineering practices: testing, CI/CD, modular design, refactoring and code quality Improve the maintainability, reliability and scalability of systems Identify and address weak engineering practices across ...

scale data pipelines, with a strong grasp of data structuring for performance and efficiencyHands-on experience with Airflow for orchestrating complex data workflows, and Apache Iceberg for table format management at scaleExtensive AWS experience including EKS, Docker, and Kubernetes; comfortable owning infrastructure deployment and scaling decisionsCI/ ...

Real‐time streaming platforms (Kafka, Redpanda, Active/RabbitMQ, etc.) SQL databases, particularly MySQL Desired Experience Experience with data warehousing tools and platforms (Snowflake, Iceberg, etc.). Familiarity with Java. Experience with cloud‐based services, particularly AWS. Proven ability to manage stakeholders, their expectations and explain complex problems ...

e.g. sampling methods), attention to detail, and a proactive empirical mindsetFamiliarity with data modelling & modern data tech stack tools such as Airflow, dbt, Kafka, Iceberg, Trino, Superset etc.Familiarity with some data visualization tools (Tableau, QlikSense, Power BI, or similar) to communicate quality insightsFamiliarity with at least one version control ...

technical solutions.Tech StackDevelopment: TypeScript, Angular/React, Python, SQL, OpenAPI.Cloud: AWS (Lambda, API Gateway, S3, SQS, SNS, Aurora, and others as required).Databases: PostgreSQL, Apache Iceberg, DynamoDB, VectorDB (e.g. ChromaDB).IaC and CI/CD: Terraform, AWS CLI, GitLab.AI Development: Agentic AI tooling (e.g. GitHub Copilot, Claude Code ...

resolving issuesHighly desirable skillsExperience with streaming frameworks such as Flink, Kafka Streams and Kafka ConnectKnowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data CatalogExposure to DataOps practices and collaboration with Data Engineering teamsFamiliarity with GPU-accelerated analytics using Spark with GPUs or RAPIDSProgramming experience with ...

delightful index consumption experience.Develop efficient, and secured cloud-native architectures.Work with relational databases (PostgresSQL), possibly NoSQL and object stores, and modern Lakehouse architectures (Iceberg)Design scalable data models, helped by peers and data governanceContribute to code reviews and be code reviewed to learn and improvePartner with global teams ...

latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information ...

throughput, low-latency software systemsExperience with pipeline orchestration frameworks (e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about ...

replication tool, and Waggle Dance , a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg , a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn ...