51 to 75 of 175 Apache Jobs in London

assistants as part of the software development lifecycle.* Experience designing, building, and operating data pipelines and ETL/ELT workflows using Databricks, dbt, and Apache Airflow.**Preferred Skills & Experience*** Experience with frameworks like ReactJS, Angular, or Vue.js to build interactive and responsive user interface.* Experience building or designing scalable ...

data generation for code or language models. Training LLMs in distributed environments. Reinforcement learning, preference optimisation or reward modelling. Data tooling such as SQL, Apache Iceberg or DuckDB. LLM‐as‐a‐judge systems or automated code evaluation. Reward‐hacking detection, robustness evaluation or safety‐focused training. Open‐source contributions ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed dataproductsand ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will be joining … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure data ...

expertise in writing complex, optimized SQL queries for data extraction and transformation.**ETL/ELT:** Hands-on experience building data pipelines using frameworks like Apache Spark, Apache Airflow, DBT, Fivetran, Matillion, or equivalent.**Databases:** Solid understanding of relational databases (e.g., PostgreSQL, MySQL) and experience with modern data warehousing ...

Warehouses Experience with enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform implementations ...

Collaborate with cross-functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial domains ...

preferred knowledge of AI infrastructure. Proficiency in programming languages such as Python, Java, or C++. Experience with data pipeline and workflow management tools (e.g., Apache Airflow, Kubeflow). Strong problem-solving skills and ability to work in a fast-paced environment. Excellent communication and collaboration skills. Proven experience ...

understanding of data management and data quality principles, with experience developing data quality checks or reports Prior experience using the Bloomberg Terminal Experience with Apache Airflow or similar tools For more information about DRW's processing activities and our use of job applicants' data, please view our Privacy Notice ...

quality, lineage, and performance across the ML lifecycle. Preferred Qualifications & Skills Strong programming background in Python. Proficiency in distributed data-processing technologies (e.g., Ray, Apache Spark, Flyte, Dask). Hands-on experience with cloud infrastructure (AWS, GCP, or Azure), Kubernetes, and distributed training frameworks (Ray, RLLib, PyTorch ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache SparkTM and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/AI models ...

data Experience with data visualization tools to create dashboards and reports Understanding of data engineering concepts (ETL, data pipelines) and automation tools such as Apache Airflow Familiarity with cloud based computing resources such as AWS Interviewing with Veeva We value your time and believe in a transparent hiring process. ...

Collaborate with cross-functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial domains ...

Linux operating systems, including command-line administration and troubleshooting. Solid experience with SQL or data pipeline technologies, Experience with data processing frameworks such as Apache Airflow, Apache Flink or Google Dataflow, with a practical understanding of ETL pipeline design and the ability to diagnose issues across data transformation …/IP networking principles, including the ability to diagnose connectivity issues and interpret network traffic. Familiarity with event streaming platforms such as Apache Kafka, including an understanding of producer/consumer patterns and how message queues are used to move data reliably between systems at scale. Familiarity with infrastructure ...

support data pipelines, ensuring they are production-ready and meet data quality standards. You will work with technologies such as Python, SQL, and Apache Spark, helping drive analytics and operational decision-making. Ideal candidates possess strong programming skills, a relevant degree, and a collaborative spirit, as you will partner ...

streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements: Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows ...

Expopsure to data governance frameworks, metadata management, and regulatory complianc It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, real-time/streaming architectures, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders ...

years' experience delivering enterprise data platforms or analytics solutions. Strong commercial experience delivering Databricks solutions. Consulting or customer-facing delivery experience. Strong understanding of Apache Spark, modern Lakehouse architecture and Unity Catalog. Experience with Azure, AWS or GCP. Experience implementing CI/CD and deployment automation. Excellent communication ...

knowledge of AI infrastructure. Good proficiency in programming languages such as Python, Java, or C++. Experience with data pipeline and workflow management tools (e.g., Apache Airflow, Kubeflow). Strong problem-solving skills and ability to work in a fast-paced environment. Excellent communication and collaboration skills. Significant experience ...

deployment. Expert-level Python and its data ecosystem (Numpy, Pandas), including designing frameworks for data tasks. Deep understanding of distributed data processing engines like Apache Spark. Ability to reason about the full data lifecycle from ingestion through to real-time serving - your system designs will need to meet downstream ...

services for personalisation · Developing Java/Spring Boot microservices with REST/gRPC APIs · Working with Kafka and event-driven architectures · Building pipelines using Apache Flink and Apache Beam · Integrating ML models and feature stores · Working with NoSQL databases and Redis caching · Using Apache Camel for integration … with stream processing concepts · Knowledge of distributed systems · Strong communication and teamwork skills · Interest in data-driven personalisation systems Nice to have · Experience with Apache Flink and Apache Beam · Experience with Kafka · Familiarity with Apache Camel · Experience with NoSQL databases (Cassandra, DynamoDB) and Redis · Cloud experience ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn ...

stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

software engineers and product managers to deliver robust data solutions. The role requires 7+ years in data engineering, including leadership, and strong expertise in Apache Spark and Azure Data Factory, with hands-on cloud experience across Azure/AWS/GCP. #J-18808-Ljbffr ...