1 to 25 of 86 Apache Iceberg Jobs in England

computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers … want to advance the capabilities of modern data systems and apply that work to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog ...

management, and quality assurance.Collaboration: Engage with business and technology teams to align the platform with organizational goals.Mandatory Skillsets: Python, Snowflake, dbt, dremio, glue jobs, Apache iceberg, AWS ,SQL.Preferred Technical Skills:Programming: 5+ years of experience in Python/Java.Cloud Expertise: Strong understanding of AWS services (e.g., Lambda, Step … Functions, ECS).Data Platforms: Hands-on experience with Snowflake and data stack technologies like Apache Iceberg and Spark.Workflow Orchestration: Exposure to tools like Apache Airflow, Prefect, Dagster, or DBT.Data Services: Familiarity with AWS Glue, Lake Formation, EMR, EventBridge, Athena, and similar services.Metadata Tools: Experience with tools like ...

semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. AI-Accelerated Schema & Pipeline … Change Data Capture (CDC), clustering, data migration, and aggregation. Use AI query-optimization tools to analyse execution plans, auto-tune complex Snowflake/Iceberg SQL workloads, and eliminate performance bottlenecks. Intelligent Telemetry & Closed-Loop Reasoning: Integrate automated AI inferencing and reasoning layers directly into data pipelines to detect telemetry ...

hybrid contract basis. This role focuses on designing, building, and optimizing modern data lakehouse architectures using AWS-native services and open table formats like Apache Iceberg. You will play a key role in enabling scalable, secure, and high-performance data platforms.Key Responsibilities Design and implement scalable AWS Data Lakehouse … solutions Develop and maintain data pipelines using Python and PySpark Work with AWS services including: S3, EMR, Lambda, IAM, EKS, MWAA Build and manage Apache Iceberg tables for efficient data storage and querying Implement CI/CD pipelines for data platform deployments Use Terraform for infrastructure as code ...

pipelines using Python and PySpark Building and optimising data workflows using AWS Glue, Athena, Redshift , and related AWS services Supporting modern lakehouse architectures, including Apache Iceberg Working with relational and NoSQL databases to ensure data is reliable and fit for purpose Collaborating with analysts, engineers, product owners … Athena, Amazon Redshift , and cloud-based data platforms Strong SQL skills and experience working with both relational and NoSQL databases Experience supporting or developing Apache Iceberg or modern lakehouse solutions Experience working in Agile delivery environments Familiarity with Linux and Git-based version control Desirable experience Experience with ...

effectively with stakeholders and champion better ways of working. Nice to Have Experience with Docker, Java/Scala, Kafka, Pulsar, Azure Event Hub, Collibra, Apache Iceberg, Apache Arrow, PySpark or AI engineering technologies such as Snowflake AI, LangChain, LlamaIndex or Pinecone would be advantageous. This ...

effectively with stakeholders and champion better ways of working. Nice to Have Experience with Docker, Java/Scala, Kafka, Pulsar, Azure Event Hub, Collibra, Apache Iceberg, Apache Arrow, PySpark or AI engineering technologies such as Snowflake AI, LangChain, LlamaIndex or Pinecone would be advantageous. This ...

customer requirements and translate them into practical technical solutions.Excellent analytical, problem-solving and communication skills.Desirable experienceData-lake or lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Snowflake or AWS Lake Formation.Image or video visualisation, enhancement, rendering or tagging applications.GIS or geo-referenced imagery.Event-driven ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline … lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development ...

stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

operating ETL/ELT pipelines and large-scale data architectures. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

operating ETL/ELT pipelines and large‐scale data architectures. Hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

environments. Technical Skills: Experience building and operating ETL/ELT pipelines. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

operating ETL/ELT pipelines and large-scale data architectures. Deep hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). ...

proactive detection and resolution of issues Collaborate with platform analytics and engineering teams to ensure efficient data modeling table design and storage optimization eg Iceberg Delta patterns Technical Qualifications 7 years of experience in a data engineering or similar role within a fast paced enterprise scale environment Bachelors … data transformation and orchestration processes Extensive Experience with AWS services including IAM Lambda EKS S3 Data Lake EMR PySpark MWAA and Lakehouse technologies, Apache Iceberg Proven expertise with Snowflake including performance tuning and efficient query design Experience with batch pipelines eg StreamSets and streaming technologies eg Kafka Nice ...

tooling, or cloud cost management across at least one major cloud provider (Azure, AWS, or GCP). Technical Skills Data engineering: strong proficiency with Apache Spark, dbt, Airflow/Azure Data Factory or equivalent orchestration, Data Lake/Apache Iceberg, and advanced SQL for large‐scale transformations. ...

working with cloud‐native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such … Glue, EMR Serverless, Lambda, and S3. Contribute to the evolution of lakehouse architecture leveraging Apache Iceberg. Apply Software Engineering Best Practices Use version control, CI/CD pipelines, automated testing, and modular code principles. Participate in pair programming, code reviews, and architectural design sessions. Data Quality & Observability Build monitoring ...

data engineering, with demonstrated expertise in designing and implementing scalable, flexible modern data architectures Experience building modern data lake/lakehouse architectures (e.g., Apache Iceberg) in cloud environments Strong knowledge of AWS (S3, Redshift) or Azure data services Experience with ETL/ELT processes and data ingestion Proven … expertise using transformation tools such as dbt Experience with data pipeline orchestration tools (such as Apache Airflow) Experience with best practices applied to data (e.g., testing, CI/CD) Advantageous Experience developing dashboards and reporting solutions using data visualization tools such as Power BI or Amazon QuickSight Experience with ...

root-cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

Experience with governance, controls, and data quality frameworks including reconciliations, lineage, auditability, and entitlements. Familiarity with lake and lakehouse table formats and concepts, including Iceberg or Delta, and columnar storage formats such as Parquet. Experience with CI/CD, infrastructure as code, containerized workloads, and DevOps practices. Experience leading ...

data engineering, with demonstrated expertise in designing and implementing scalable, flexible modern data architectures Experience building modern data lake/lakehouse architectures (e.g., Apache Iceberg) in cloud environments Strong knowledge of AWS (S3, Redshift) or Azure data services Experience with ETL/ELT processes and data ingestion Proven … expertise using transformation tools such as dbt Experience with data pipeline orchestration tools (such as Apache Airflow) Experience with best practices applied to data (e.g., testing, CI/CD) Advantageous Experience developing dashboards and reporting solutions using data visualization tools such as Power BI or Amazon QuickSight Experience with ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root-cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark. Working knowledge of common data formats such as JSON, Avro and Parquet. Technology EnvironmentThe role will involve working with a modern and evolving … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, KafkaData formats: JSON, Avro, ParquetPlatforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQEngineering and deployment: CI/ ...

Catalog, Spark, Structured Streaming, and Lakehouse architectureStrong hands-on experience with Snowflake, including data modelling, performance optimisation, Snowpark, data sharing, and governance capabilitiesExpertise in Apache Spark, Python, SQL, and modern ELT/ETL frameworksProven ability to rapidly evaluate, prototype, and compare emerging technologies and architectural patternsExperience designing large-scale … market microstructureExperience with AI/ML platforms, feature stores, and MLOps frameworksExperience evaluating alternative analytical databases such as DuckDB, ClickHouse, Redshift, BigQuery, KDB+, or Apache Iceberg-based architecturesExperience with data catalogue and governance platformsExposure to modern visualisation and BI tools such as Power BI, Tableau, Sigma, or ThoughtSpotExperience ...