201 to 225 of 238 Apache Jobs in London

architecture, acting as the design authority to ensure secure, scalable, auditable solutions across classified environments. The position requires extensive experience with AWS data platforms, Apache NiFi, API #J-18808-Ljbffr ...

experience writing high-quality production code Strong foundation in operating systems and systems programming Proven track record in performance engineering Familiarity with tools like Apache Arrow, Parquet, DataFusion, Clickhouse, or DuckDB is a plus Understanding of cutting-edge ML infrastructure stack (e.g. PyTorch, CUDA) is also a plus Experience ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern observability ...

software development lifecycleSolid background in Java/Scala and a scripting language like Python. Experience building large scale data pipelinesStrong familiarity with the Apache Hadoop ecosystem including : Spark, Kafka, Flink, Iceberg/Delta Lake/Hive, Apache Presto/Trino, etc. Experience with relational SQL and NoSQL databases ...

design, develop, deploy, and optimize scalable data solutions on Microsoft Azure. The ideal candidate will bring strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, Python, and SQL, with proven experience building production-grade data pipelines, implementing governance through Unity Catalog, and automating deployments using GitLab-based …/CD and Databricks Asset Bundles. Key Responsibilities Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design and manage ...

data strategy Helping build and mentor the future data engineering team as the company scales Python, PostgreSQL, MongoDB, vector databases (Qdrant, Milvus or pgvector), Apache Spark, Apache Airflow, Kafka, Elasticsearch/OpenSearch, Pandas, Polars. What they're looking for: 7+ years of experience in data engineering or backend ...

governance frameworks covering data ownership, stewardship, classification (PII, sensitive, public), and retention policies Establish data lineage and cataloguing standards, working with tooling such as Apache Atlas, Collibra, or Snowflake Horizon Catalog Design and enforce data quality rules and SLAs at ingestion, transformation, and consumption layers Collaborate with the Risk … documentation, incremental strategies Application Databases Data Integration AWS – S3, RDS, Aurora, Redshift (migration context), Glue, Lake Formation, IAM, VPC Data Governance Data lineage, cataloguing (Apache Atlas/Collibra/Snowflake Horizon), GDPR, BCBS 239, MDM AI/ML Data Query & Performance SQL optimisation, clustering keys, partitioning, query profiling, cost ...

cloud-scale storage systems. Ideal candidates should have over 5 years of experience and a strong background in operating systems. Familiarity with tools like Apache Arrow and an understanding of machine learning infrastructure are preferred. The role requires in-person work in either NYC or London, supporting innovative storage ...

management, and quality assurance.Collaboration: Engage with business and technology teams to align the platform with organizational goals.Mandatory Skillsets: Python, Snowflake, dbt, dremio, glue jobs, Apache iceberg, AWS ,SQL.Preferred Technical Skills:Programming: 5+ years of experience in Python/Java.Cloud Expertise: Strong understanding of AWS services (e.g., Lambda, Step Functions … .Data Platforms: Hands-on experience with Snowflake and data stack technologies like Apache Iceberg and Spark.Workflow Orchestration: Exposure to tools like Apache Airflow, Prefect, Dagster, or DBT.Data Services: Familiarity with AWS Glue, Lake Formation, EMR, EventBridge, Athena, and similar services.Metadata Tools: Experience with tools like Amundsen, Atlas, DataHub ...

Solid knowledge of Linux/Unix, OpenShift/Kubernetes, and Ansible. Distributed Systems: Experience building and supporting reactive, event-sourced systems, with hands-on Apache Spark and Apache Flink. APIs & Data Formats: Strong understanding of Apache HTTP, Envoy, and REST API design, plus experience with Protobuf, Avro ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply for full ...

needed — even if hands‐on coding isn’t a part of the day‐to‐day Hands‐on experience with distributed event streaming systems like Apache Kafka. Familiarity with OLAP databases such as Apache Pinot or ClickHouse Proficient in modern data lake and warehouse tools such as S3, Databricks … Snowflake Strong foundation in the .NET ecosystem, container orchestration with Kubernetes, and cloud platforms, especially AWS Experience with distributed data processing engines like Apache Flink or Apache Spark is nice to have EPIC JOB + EPIC BENEFITS = EPIC LIFE We pay 100% for benefits except ...

basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack. You will work within a government-aligned, secure environment, ensuring data remains hosted ...

Development Contribute hands-on as a primary engineer utilizing our modern tech stack: Core Java (multi-threading, concurrency, memory optimization),CouchbaseDB, Kafka, and Apache Storm. Solve complex real-time risk calculation requirements , handling heavy stream processing and highly concurrent data flows. Required Experience extensive development experience , with a strong … including multi-threading, memory management, garbage collection tuning, and concurrency utilities. Solid experience with real-time stream processing and distributed data architectures (e.g., Kafka, Apache Storm,CouchbaseDB/NoSQL, Spring). Familiarity with modern infrastructure paradigms , such as containerization (OpenShift/Kubernetes) and cloud platforms (GCP/ ...

Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related … between backend/cloud development and data platform engineering roles* 5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP* 5+ years of experience programming with Java, Scala or Kotlin.* Proven experience owning robust ...

Familiarity with PostgreSQL and Snowflake Preferred: Familiarity with Web Frameworks such as Django, Flask or FastAPI Preferred: Familiarity with event streaming platforms such as Apache Kafka Preferred: Familiarity with data pipeline platforms such as Apache Airflow Preferred: Familiarity with Java Benefits and Perks We value the total wellbeing ...

/month in the officeWe’re looking for a **Confluent Solution Architect** to lead the design and architecture of enterprise-grade **streaming solutions** with **Apache Kafka** and the **Confluent Platform**.**What you’ll do:*** Design scalable, secure, and resilient data streaming ecosystems* Translate business requirements into technical solutions … Lead client engagements and guide implementation teams* Integrate Kafka with enterprise systems, cloud platforms, and microservices**Must-Have Skills:** 8+ years in Apache Kafka & Confluent Platform Deep knowledge of Kafka internals & KRaft architecture Experience with Confluent Cloud, stream governance, RBAC, and data lineage Expertise in stream processing (Apache ...

sprint planning, and review meetings to ensure continuous communication and progress tracking. Design, build, and optimize scalable data pipelines using Azure Data Factory and Apache Airflow for scheduling and orchestration. Implement efficient data ingestion, transformation, and movement processes from various sources into Snowflake Data Warehouse. Develop reusable components … solutions. Proven experience in building and maintaining data pipelines and workflows. Strong knowledge of Azure Data Factory for data movement and transformation. Proficiency with Apache Airflow (orchestration, scheduling, DAGs, error handling). 5+ Years experience in Data Engineering using Azure Data Factory, Snowflake and AirflowTravel: There could ...

Requirements: Proven commercial experience working as a Data Architect within the Defence space. Active DV Clearance. Strong AWS data engineering experience Hands-on Apache NiFi expertise Experience delivering secure ETL/ELT and data integration pipelines Ability to lead technical delivery teams and mentor engineers Strong stakeholder management … communication skills AWS data platforms and secure engineering Apache NiFi pipeline development and support API, file-based, legacy and bespoke integrations Data mapping, transformation and validation CI/CD, DevSecOps and release management Testing (unit, integration, performance and security) Monitoring, logging, alerting and operational support Nice to have ...

Requirements: Proven commercial experience working as a Data Architect within the Defence space. Active DV Clearance. Strong AWS data engineering experience Hands-on Apache NiFi expertise Experience delivering secure ETL/ELT and data integration pipelines Ability to lead technical delivery teams and mentor engineers Strong stakeholder management … communication skills AWS data platforms and secure engineering Apache NiFi pipeline development and support API, file-based, legacy and bespoke integrations Data mapping, transformation and validation CI/CD, DevSecOps and release management Testing (unit, integration, performance and security) Monitoring, logging, alerting and operational support Nice to have ...

using JavaScript TypeScript HTML5 CSS3 and state management tools like Redux or Context API Integrate Kafka for event streaming and realtime data processing Utilize Apache Ignite for inmemory data grid and caching solutions Collaborate with crossfunctional teams to gather and analyze business requirements ideally within the investment banking … support CICD pipelines and automate deployments using Kubernetes and AWS Mentor and provide technical guidance to junior developers on Core Java ReactJS Kafka Apache Ignite and cloudnative best practices Participate actively in Agile ceremonies including sprint planning daily standups and retrospectives Ensure adherence to security compliance standards and best ...

Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling. What You'll Do: Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin. Proven experience utilizing AI technologies ...

Familiarity with PostgreSQL and Snowflake Preferred: Familiarity with Web Frameworks such as Django, Flask or FastAPI Preferred: Familiarity with event streaming platforms such as Apache Kafka Preferred: Familiarity with data pipeline platforms such as Apache Airflow Preferred: Familiarity with Java Preferred: Experience in one or more relevant financial ...