26 to 46 of 46 Apache Iceberg Jobs in London

modelling, medallion architecture) Leading the design of cloud data platforms (AWS) – including services like Redshift, S3, Glue, Athena Evaluating and introducing new technologies (e.g. Iceberg, Delta Lake, observability tools) Driving data governance, standards, and best practice Acting as a trusted advisor to both technical and business stakeholders What ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt)Experience designing and operating orchestration pipelines using Airflow ...

native serverless data platform. Data teams write plain Python, and we handle packaging, dependency resolution, scheduling, and execution across a serverless runtime built on Apache Iceberg. There's no clusters to manage, no YAML to write, no pipelines to rewrite to fit someone else's engine. … senior team split between London and Berlin. We've built a lot of tech: Custom run dispatch service, multi-region routing, storage based on Apache Iceberg, and the runtime itself, to name a few. This is deep systems work—things like scheduling, distributed execution, and performance. ...

Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well as batch and streaming processing Knowledge of data quality, governance, cataloguing and observability tools (e.g. Datadog), with ...

Understanding of Azure data services and API-based integration. Next.js 14, FastAPI Python, Pipeline orchestration, Data transformation, Data quality, Casbin RBAC, Kuzu knowledge graph, Apache Iceberg, MCP AI server integration and ODCS v2 data contracts layer. Experience in establishing the infrastructure, pipelines, processes, and governance required for production ...

Amazon S3 AWS Lake Formation Amazon Athena Experience building Lakehouse architectures and modern data platforms. Experience working with data formats such as Parquet and Iceberg tables. Strong experience with Terraform for infrastructure automation. #J-18808-Ljbffr ...

users, observing where they struggle, and turning that into product improvementsWhat will set you apartHands-on experience with the modern data stack, for example Iceberg or Delta, Snowflake or Databricks, dbt or AirflowExperience with a data migration or re-platforming project, on either the product or the engineering sideExperience ...

implementing computational improvements Making improvements and/or fixing bugs by submitting patches to open-source projects in our stack, such as Spark, Iceberg, Hadoop, Airflow, and many others Acting as an expert on Spark and Scala Mentoring engineers (including through pairing, design discussions, and knowledge-sharing sessions ...

data quality, governance and platform delivery.Understanding of data product concepts, metadata, lineage, access control or platform operating models would be advantageous.Familiarity with Snowflake, Databricks, Apache Iceberg or similar modern data platform technologies is beneficial.Formal training in Product Management, Product Ownership or Agile delivery is helpful but not essential.Risk ...

decisions that matter: architecture, system design, and the tradeoffs that determine whether a platform scales gracefullyNice-to-haveLakehouse and streaming technologies — experience with Databricks, Apache Iceberg, Kafka, or similar is a strong advantageWhat we offerDiverse and internationally distributed team: joining our team means becoming part of a large ...

compliance colleaguesWhat will set you apartExperience in financial services, payments or another regulated industryHands-on familiarity with the modern data stack, for example Iceberg or Delta, Snowflake or Databricks, and dbtFamiliarity with AI governance frameworks such as the EU AI Act, DORA or MAS FEATAdditional InformationFind out moreHow ...

familiar withFX or financial markets experienceTerm structure modelling, stochastic calculus or Monte Carlo methodsInterest rate curve bootstrappingAlgorithmic execution experienceData lake or warehouse experience (Snowflake, Iceberg, Spark etc.)We’re people without borders — without judgement or prejudice, too. We want to work with the best people, no matter their background. ...

peers on technical direction within your teamRequired Skills and ExperienceStrong analytic skills related to working with unstructured datasetsPython (PySpark, Pandas, PyArrow)Distributed data processing (Apache Spark)Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)Cloud services (AWS, Azure or GCP)Messaging/Streaming (Kafka … Other Cloud Queuing Native services)SQL and NoSQL databases and storage (HDFS, Iceberg, Elastic, S3, Data Lake)Containerisation and orchestration (Docker/Kubernetes/Openshift)Testing frameworks and best practicesWe appreciate you may not be an expert in every area above - we can support with training and development ...

pipelines and streaming architectures. Establish best practices for monitoring, reliability, and scale across our data ecosystem (using Python, dbt, Airflow, Kafka, and Trino/Iceberg).Build the Unified Customer View: Productionize PoCs into scalable Airflow/dbt data workflows to lay the groundwork for our Customer Data Platform … Advanced proficiency in Python and proven experience architecting, deploying, and maintaining Big Data and streaming/batch pipelines (e.g., Kafka Streams, Event Streaming, Trino, Iceberg).dbt & Airflow Proficiency: Hands-on experience using dbt for scalable data transformations and Airflow for workflow orchestration.Data Architecture Mastery: A strong background in designing ...

Data Engineer IIAre you passionate about building scalable, high-performance data-driven applications Do you enjoy turning complex datasets into robust, reliable systems that deliver real business impact About the team:The Search and AI ...

project stakeholders; Strong problem-solving skills; Ability to refine problem statements and propose solutions considering effort, impact, and scalability trade-offs; Nice to have: Apache Spark, Iceberg, Kafka, dbt, Scikit-Learn, XGBoost, PyTorch, MLFlow, GraphFrames, Ray, AWS S3, EMR, SageMaker, Lakeformation, Terraform, Docker, GitHub CI/CD, Knowledge ...

ability to help refine problem statements and propose solutions taking effort-impact-scalability tradeoff into account.Some skills that will make you stand out: Apache Spark, Iceberg, Kafka, dbtScikit-Learn, XGBoost, PyTorch, MLFlow,, GraphFrames, RayAWS (S3, EMR, SageMaker, Lakeformation), Terraform, Docker, GitHub CI/CDKnowledge Graphs (+ RAG), graph ...

software development lifecycleSolid background in Java/Scala and a scripting language like Python.Experience building large scale data pipelinesStrong familiarity with the Apache Hadoop ecosystem including : Spark, Kafka, Flink, Iceberg/Delta Lake/Hive, Apache Presto/Trino, etc.Experience with relational SQL and NoSQL databases, including ...

Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related … between backend/cloud development and data platform engineering roles* 5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP* 5+ years of experience programming with Java, Scala or Kotlin.* Proven experience owning ...

microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

strong GCP experience. The role will focus on building scalable data platforms and pipelines on GCP, working with technologies including Kafka, Python, Spark and Iceberg , while implementing robust data governance, lineage and auditability. Key Requirements: Active SC Clearance - (SC Elapsed Candidates will also be considered!) Strong GCP experience Kafka … Python, Spark & Iceberg Data Governance and Lineage expertise Please note that rate and IR35 determination are still being finalised and are expected next week . Given the project timelines, immediately available contractors are strongly preferred .Please apply now to avoid missing out on being shorlisted for this. ...