24 of 24 Remote/Hybrid Permanent Apache Iceberg Jobs

computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. We build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, bringing advances in open-source engines and table formats into the demanding environments our customers … want to advance the capabilities of modern data systems and apply that work to consequential problems. Technologies We Use Java, Scala, Rust, and Python Apache Spark, Apache DataFusion, Apache Comet, and Velox for data processing and query execution Apache Iceberg for table management and catalog ...

analytical and operational domains.Establish ELT/ETL and streaming architectures on Snowflake Snowpipe, Snowpipe Streaming, Dynamic Tables, Streams and Tasks, external tables, and Apache Iceberg tables along with orchestration standards (dbt, Airflow, Snowflake Tasks) and CI/CD/DataOps best practices.Implement security-by-design in Snowflake: RBAC … Azure, or GCP) including Snowflake deployment, networking (Private Link), and cloud storage integration (S3, ADLS, GCS); familiarity with lakehouse stacks (Databricks, Delta Lake, Apache Iceberg) is a plus.Mastery of Snowflake security and governance: RBAC, tag-based governance, dynamic data masking, row access policies, column-level security, Tri-Secret ...

semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. AI-Accelerated Schema & Pipeline … Change Data Capture (CDC), clustering, data migration, and aggregation. Use AI query-optimization tools to analyse execution plans, auto-tune complex Snowflake/Iceberg SQL workloads, and eliminate performance bottlenecks. Intelligent Telemetry & Closed-Loop Reasoning: Integrate automated AI inferencing and reasoning layers directly into data pipelines to detect telemetry ...

LilEngine is leading the Lakehouse, a company-strategic effort to make Apache Iceberg a first-class citizen in Spotify's data ecosystem. We're building it in close collaboration with Google, on open standards, so that different kinds of data are queryable the same way and teams … bring the right engine, from BigQuery to Dataflow to Trino, to the job. The work has two parts. Hecate enables Iceberg for the Parquet datasets that already exist across Spotify, so teams get the benefits without rewriting their data. Alongside it, we're building the foundations ...

operating ETL/ELT pipelines and large‐scale data architectures. Hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

operating ETL/ELT pipelines and large-scale data architectures. Deep hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

environment Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Experience working with lakehouse/medallion architectures in Databricks Experience with experimentation support tooling (such as Optimizely) A genuine curiosity about ...

Amazon S3 AWS Lake Formation Amazon Athena Experience building Lakehouse architectures and modern data platforms. Experience working with data formats such as Parquet and Iceberg tables. Strong experience with Terraform for infrastructure automation. #J-18808-Ljbffr ...

transparency. Skills & Experience Essential Proven in-depth commercial experience in the following key areas: Databricks- Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning. Apache Spark & Scala- High-performance distributed data processing and optimisation. Cloud Data Engineering (AWS) - Cloud-native data platforms and distributed computing. Data Services Architecture & Engineering … knowledge of client core business functions Demonstrated leadership, project management, and development skills Relationship and consensus building skills Preferred Kafka/Event-Driven Architecture Apache Iceberg/Delta Lake Python AI/ML & LLM integration Financial Services/Capital Markets Education: Bachelor's degree/University degree ...

Tool (DBT) with a demonstrated ability indeveloping data models, contracts, tests, validation, and transformations Experienceworking with modern data distributed file formats (i.e., Parquet, Delta,Iceberg, Hudi) Demonstratedexperience with building data ingestion pipelines from REST API data sources. Strongability to produce technical documentation that can be understood by bothtechnical ...

Lead Data EngineerManchester | Hybrid | Circa £85K An established organisation is looking for a hands-on Lead Data Engineer to take ownership of its existing data estate and lead the development of a modern, scalable data ...

Data Engineering Lead – AWS & AI - Manchester An established financial services organisation is looking for a hands-on Data Engineering Lead to take ownership of its existing data estate and deliver a modern platform supporting Business ...

Data Engineering Lead AWS & AI - Manchester An established financial services organisation is looking for a hands-on Data Engineering Lead to take ownership of its existing data estate and deliver a modern platform supporting Business ...

streaming workloads, managing schemas as they evolve, and guaranteeing that consumers only read complete data. We build on open standards like OpenLineage and Apache Iceberg, and define the internal standards for naming, storage and metadata that the rest of Spotify builds against. As the platform takes ...

Rate: £650 - £750 Location: Hybrid – 2/3 days per week on-site Contract: 6 months | Outside IR35 Key Skills: Data Lakehouse PySpark DBT Iceberg AWS Strong data engineering and pipeline development experience You’ll be responsible for designing, building and optimising scalable data pipelines and lakehouse solutions within ...

align solutions with strategic goals. The role requires 5+ years in Python/Java, strong AWS knowledge, and hands-on experience with Snowflake, Iceberg, Spark, and DBT. The position offers a hybrid work arrangement and involvement in enterprise-grade data projects. #J-18808-Ljbffr ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka … Other Cloud Queuing Native services) SQL and NoSQL Storage (HDFS, Iceberg, Elastic, S3, Data Lake) Containerisation and orchestration (Docker/Kubernetes/Openshift) Testing frameworks and best practices Use of Code repositories, familiarity with branching strategies, pull requests and merge processes Benefits £: Competitive salary Contributory Pension Scheme ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka … Other Cloud Queuing Native services) SQL and NoSQL Storage (HDFS, Iceberg, Elastic, S3, Data Lake) Containerisation and orchestration (Docker/Kubernetes/Openshift) Testing frameworks and best practices Use of Code repositories, familiarity with branching strategies, pull requests and merge processes We appreciate you may not be an expert ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka … Other Cloud Queuing Native services) SQL and NoSQL Storage (HDFS, Iceberg, Elastic, S3, Data Lake) Containerisation and orchestration (Docker/Kubernetes/Openshift) Testing frameworks and best practices Use of Code repositories, familiarity with branching strategies, pull requests and merge processes We appreciate you may not be an expert ...

team of highly technically skilledengineerswhoaredesigningCreditsafe’snewConsumer Bureauwithdata quality,lineage, andauditabilityas primary goals whilstmaintaininghigh throughputand scalability.Thedata platformwill be builtusingAWSinfrastructure such as S3and S3-tables(Iceberg), AuroraServerless, DynamoDB, Lambda,Fargate, ECS, Airflow,Sparkand Kinesis.Theplatformisexpectedto manage overone hundred millionobjectswith a high volume of daily updateshandlingaddition, deletion,andcorrectionof our dataand indexesinanauditable manner.The platform … Amigo's refinement sessions all the way to post-deployment observability. Design and build metadata driven,eventbaseddistributeddata processing platformusing technologies such asPython,Spark,Airflow, Iceberg,DynamoDB,AWS Glue, S3. Playan activerole in the design, development, and deployment ofour business-criticalsystem. Building and scalingCreditsafe’sdata pipelinescombining batch and streaming approachesandtargetingsub ...

Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling. What You'll Do Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related … between backend/cloud development and data platform engineering roles 5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP 5+ years of experience programming with Java, Scala or Kotlin. Proven experience utilizing ...

Position: App Dev - Java/Scala/Pyspark - Lead W2 only Location: Wilmington, Delaware Duration: Contract Job ID: 180176 Job Overview: We are seeking a skilled and experienced Back-End Developer with expertise in Java ...