26 to 50 of 199 Apache Jobs in London

pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift or equivalent ...

Oracle databases. Experience designing and supporting ETL/ELT pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of Autosys and Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift ...

data pipeline development, ETL/ELT processes, and data warehousing solutions. Proficient with cloud platforms (AWS, GCP, Azure) and big data tools like Apache Spark, Python, etc. Experience with relational and NoSQL databases. Solid understanding of data modeling, data architecture, engineering frameworksProven ability to lead and scale engineering teams ...

with data transformation tools such as dbt, or Dataflow. Understanding of data modeling techniques. Experience with using visualisation platforms such as Looker, Tableau, or Apache Superset. Understanding of software engineering best practices and their application to data processing systems. Knowledge of Python, Java or Flink is a plus ...

self-motivated, inquisitive mindset and a methodical, agile approach to problem-solving Solid Linux administration and networking skills, plus Git-based workflows Familiarity with Apache Airflow, Dask and JupyterHub Experience with modern build systems (especially Bazel) and CI/CD pipelines. What we offer At Macquarie, you’re empowered ...

mesh concepts and their practical implementation within large organisations Experience integrating Databricks with AWS services including S3, IAM, VPC, and AWS Glue Familiarity with Apache Spark, dbt, data pipelines, and modern analytics architectures Understanding of access management, audit frameworks, and enterprise governance controls Delivery Experience Experience leading data product ...

Databricks REST APIs, Prometheus, or Azure Monitor. Maintain best practices for data engineering performance, testing, and documentation. Technical Skills Required Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake. Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies. Experience tuning job configurations (e.g. ...

Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python ...

experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

have GCP (e.g. IAM, Cloud Run, GKE, Cloud Storage, VPC networking) Snowflake Data Platform Terraform Gitlab CI Familiarity with SQL Docker and Kubernetes Apache Airflow Observability tools – especially those relevant to Data Platforms DBT Informatica. Salary & Benefits Pay: £62,900.00 - £115,000.00 Annual Contract Type: Permanent Hours of Work ...

knowledge of optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow … cost optimisation strategies for cloud data platforms Experience with data quality frameworks and implementation Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Experience with ...

skills; experience with Oracle databases . Experience designing and supporting ETL/ELT pipelines and data quality frameworks . Working knowledge of Autosys& Apache Airflow Experience with CI/CD tools (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift) Familiarity with AWS S3 for large-scale data storage. Experience ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data Meshes ...

production problems and taking responsibility for critical infrastructure. Startup Mindset: Proactive, resourceful and comfortable taking ownership in a fast-moving engineering environment. Bonus : with Apache Airflow, real-time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial-market systems. Benefits An Ambitious, Collaborative Culture: Join ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open‐source frameworks Familiarity with AWS‐native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance practices ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

written fluency in English is required — we're a distributed team and written communication is important. Preferred Qualifications Airflow for orchestration. Experience with Apache Hudi. Experience with financial, transactional, or payment data. What We Offer at Yuno Competitive Compensation. Remote Work – You can work from everywhere! Home Office Bonus ...

legal advisors to ensure holistic solutions Your areas of knowledge and expertise Proven experience designing and implementing data lakehouse architectures (Databricks, Delta Lake, Apache Flink, or similar technologies) Strong, hands-on AWS cloud experience including services such as S3, Glue, Lake Formation, Athena, and Redshift Demonstrated expertise ...

validate transformations and trace data across platforms. Tooling/Delivery Methods Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift ...

Robust knowledge of data engineering, including data modelling, ETL/ELT, pipeline orchestration, testing, and data quality management.* Hands-on experience with Databricks and Apache Spark to build and operate scalable data pipelines and analytics or AI workflows.* Proven proficiency in applied AI, demonstrated through delivered solutions using machine ...

with Kubernetes-native platforms to deploy and operate data workloads across cloud environments is preferred. Knowledge of Golang would be an advantage. Familiarity with Apache Iceberg table format and schema management via catalogs such as AWS Glue, with an appreciation for data quality and consistency is a strong advantage. ...

into a coherent operational story. Data engineering & processing systems: You have built, operated, or supported production data pipelines using distributed processing systems such as Apache Spark or Databricks, and understand common failure modes, performance tradeoffs, and operational challenges in batch and hybrid pipelines. Data platforms & analytics systems: You have ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

expertise in writing complex, optimized SQL queries for data extraction and transformation. ETL/ELT: Hands-on experience building data pipelines using frameworks like Apache Spark, Apache Airflow, DBT, Fivetran, Matillion, or equivalent. Databases: Solid understanding of relational databases (e.g., PostgreSQL, MySQL) and experience with modern data warehousing ...