1 to 25 of 57 Remote Apache Iceberg Jobs

computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers … want to advance the capabilities of modern data systems and apply that work to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog ...

would be advantageous. Experience with Atlan and Anomalo preferred. Utilising Cloud Agnostic Technologies: Experience designing and implementing solutions using open ecosystem technologies. Knowledge of Apache Iceberg, Apache Spark, Apache Kafka, and open table formats is highly desirable. Benefits and perks There's one thing people ...

business. This role is ideal for someone with strong AWS experience and a background working with modern data tooling including dbt, PySpark, Airflow, Redshift, Iceberg, OpenSearch, Knowledge Graphs, Python, and CircleCI . Key Responsibilities Design and develop scalable ETL/ELT data pipelines using Python and PySpark Build … maintain data transformation frameworks using dbt Develop and orchestrate workflows using Apache Airflow Deliver high-performance data warehouse solutions within Amazon Redshift Support the implementation and optimisation of Apache Iceberg lakehouse architectures Develop and enhance search capabilities using OpenSearch Work with Knowledge Graph technologies to improve data ...

analytical and operational domains.Establish ELT/ETL and streaming architectures on Snowflake Snowpipe, Snowpipe Streaming, Dynamic Tables, Streams and Tasks, external tables, and Apache Iceberg tables along with orchestration standards (dbt, Airflow, Snowflake Tasks) and CI/CD/DataOps best practices.Implement security-by-design in Snowflake: RBAC … Azure, or GCP) including Snowflake deployment, networking (Private Link), and cloud storage integration (S3, ADLS, GCS); familiarity with lakehouse stacks (Databricks, Delta Lake, Apache Iceberg) is a plus.Mastery of Snowflake security and governance: RBAC, tag-based governance, dynamic data masking, row access policies, column-level security, Tri-Secret ...

semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. AI-Accelerated Schema & Pipeline … Change Data Capture (CDC), clustering, data migration, and aggregation. Use AI query-optimization tools to analyse execution plans, auto-tune complex Snowflake/Iceberg SQL workloads, and eliminate performance bottlenecks. Intelligent Telemetry & Closed-Loop Reasoning: Integrate automated AI inferencing and reasoning layers directly into data pipelines to detect telemetry ...

Snowflake. Experience in scientific, bioinformatics, computational biology, immunology, clinical, healthcare, or other regulated data environments. Experience with lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Athena, Glue, Trino, Spark, or DuckDB. Experience with regulated software, security, quality, or healthcare frameworks such ...

customer requirements and translate them into practical technical solutions.Excellent analytical, problem-solving and communication skills.Desirable experienceData-lake or lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Snowflake or AWS Lake Formation.Image or video visualisation, enhancement, rendering or tagging applications.GIS or geo-referenced imagery.Event-driven ...

translate them into practical technical solutions. Excellent analytical, problem-solving and communication skills. Desirable experienceData-lake or lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Snowflake or AWS Lake Formation. Image or video visualisation, enhancement, rendering or tagging applications. GIS or geo-referenced imagery. ...

operating ETL/ELT pipelines and large‐scale data architectures. Hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

operating ETL/ELT pipelines and large-scale data architectures. Deep hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

root-cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

based data platform. Key Skills Required Strong AWS data engineering experience Hands-on expertise with DBT and PySpark Proven Python development skills Experience with Apache Airflow for orchestration Strong knowledge of Amazon Redshift Experience with Apache Iceberg and modern data lake architectures Familiarity with OpenSearch for search ...

environment Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Experience working with lakehouse/medallion architectures in Databricks Experience with experimentation support tooling (such as Optimizely) A genuine curiosity about ...

scale data pipelines, with a strong grasp of data structuring for performance and efficiencyHands-on experience with Airflow for orchestrating complex data workflows, and Apache Iceberg for table format management at scaleExtensive AWS experience including EKS, Docker, and Kubernetes; comfortable owning infrastructure deployment and scaling decisionsCI/ ...

structured streaming, Delta Lake and Delta Live Tables Proven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache Iceberg Proficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP) Hands-on experience with CI/… security practices and enterprise integration controls Background in financial trading data or related domains Official Databricks certifications such as Certified Data Engineer Professional or Apache Spark Developer We offer EPAM Employee Stock Purchase Plan (ESPP) Protection benefits including life assurance, income protection and critical illness cover Private medical insurance ...

Tool (DBT) with a demonstrated ability indeveloping data models, contracts, tests, validation, and transformations Experienceworking with modern data distributed file formats (i.e., Parquet, Delta,Iceberg, Hudi) Demonstratedexperience with building data ingestion pipelines from REST API data sources. Strongability to produce technical documentation that can be understood by bothtechnical ...

transparency. Skills & Experience Essential Proven in-depth commercial experience in the following key areas: Databricks- Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning. Apache Spark & Scala- High-performance distributed data processing and optimisation. Cloud Data Engineering (AWS) - Cloud-native data platforms and distributed computing. Data Services Architecture & Engineering … knowledge of client core business functions Demonstrated leadership, project management, and development skills Relationship and consensus building skills Preferred Kafka/Event-Driven Architecture Apache Iceberg/Delta Lake Python AI/ML & LLM integration Financial Services/Capital Markets Education: Bachelor's degree/University degree ...

Amazon S3 AWS Lake Formation Amazon Athena Experience building Lakehouse architectures and modern data platforms. Experience working with data formats such as Parquet and Iceberg tables. Strong experience with Terraform for infrastructure automation. #J-18808-Ljbffr ...

streaming workloads, managing schemas as they evolve, and guaranteeing that consumers only read complete data. We build on open standards like OpenLineage and Apache Iceberg, and define the internal standards for naming, storage and metadata that the rest of Spotify builds against. As the platform takes ...

index consumption experience. Develop efficient, and secured cloud-native architectures. Work with relational databases (PostgresSQL), possibly NoSQL and object stores, and modern Lakehouse architectures (Iceberg)Design scalable data models, jointly with data governanceContribute to code reviews, mentor peers and more junior members of the team, and help shape ...

Lead Data Engineer Manchester | Hybrid | Circa £85K An established organisation is looking for a hands-on Lead Data Engineer to take ownership of its existing data estate and lead the development of a modern, scalable ...

Lead Data EngineerManchester | Hybrid | Circa £85K An established organisation is looking for a hands-on Lead Data Engineer to take ownership of its existing data estate and lead the development of a modern, scalable data ...

Data Engineering Lead – AWS & AI - Manchester An established financial services organisation is looking for a hands-on Data Engineering Lead to take ownership of its existing data estate and deliver a modern platform supporting Business ...