76 to 100 of 413 Apache Jobs in the UK

Layer: Design and implementation for governed reusable data views. Data Governance Tools: Collibra, Alation, Talend or equivalent. Expertise in ETL tools and processes (e.g., Apache NiFi, Talend, Informatica)Security: Data Privacy, Data Protection & Encryption frameworks & governanceEnterprise Architecture Frameworks: TOGAF, Zachman; familiarity with ITIL processes. Additional Helpful Skills: Cloud Platform ...

networking CI/CD The Atlassian Suite (JIRA, Confluence, Bitbucket, Bamboo) Azure Services Machine learning algorithms Big Data (Hadoop, Azure Data Lake, S3, Apache Spark) Additional programming languages – Python, Scala, Typescript, CSS 3/SASS (or other pre-processors) Other Information The successful candidate will be rewarded with competitive ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and "infrastructure as a code" tools ...

project stakeholders; Strong problem-solving skills; Ability to refine problem statements and propose solutions considering effort, impact, and scalability trade-offs; Nice to have: Apache Spark, Iceberg, Kafka, dbt, Scikit-Learn, XGBoost, PyTorch, MLFlow, GraphFrames, Ray, AWS S3, EMR, SageMaker, Lakeformation, Terraform, Docker, GitHub CI/CD, Knowledge Graphs ...

critical data ecosystem while supporting the delivery of high-value data products. Key Responsibilities Design, build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimise batch and streaming data workflows, ensuring reliability, scalability and performance. Contribute to the design and implementation of modern Lakehouse data ...

ensure data accuracy and reliability. Required Technical SkillsStrong proficiency in Python. Experience with monitoring and observability tools. Hands-on experience with workflow orchestration tools - Apache Airflow. Familiarity with data pipelines and ETL processes. Proficiency with containerization and orchestration technologies - Docker, KubernetesStrong understanding of CI/CD pipelines and toolsStrong ...

production problems and taking responsibility for critical infrastructure. Startup Mindset: Proactive, resourceful and comfortable taking ownership in a fast-moving engineering environment. Bonus : with Apache Airflow, real-time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial-market systems. Why Permutable Real Ownership: Take significant ownership ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

root‐cause analysis Experience building or supporting production data pipelines in a collaborative engineering environment Experience with distributed data processing frameworks such as Apache Spark Working knowledge of JSON, Avro and Parquet Nice to have: technical design ownership across multiple datasets or pipeline domains, experience guiding implementation standards ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

written fluency in English is required — we're a distributed team and written communication is important. Preferred Qualifications Airflow for orchestration. Experience with Apache Hudi. Experience with financial, transactional, or payment data. What We Offer at Yuno Competitive Compensation. Remote Work – You can work from everywhere! Home Office Bonus ...

build out technology roadmapsCodebase ownershipTechnical Application ownershipQualificationsStrong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , HarnessStrong experience of developing on Apache Spark/HadoopStrong experience with AWS cloud services including AWS Glue, S3, StepStrong experience building data pipelines and data lakeExperience of the full development ...

validate transformations and trace data across platforms. Tooling/Delivery Methods Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift ...

time streaming infrastructureOwn the architecture and roadmap for high-volume low-frequency data systems, with focus on data stack like Spark, Kafka, Iceberg, RisingWave, Apache FlinkDesign and operate scalable data architecture that serve trading, risk, compliance, analytics and many product teams. Drive adoption of AI automation and intelligent workflows ...

Modern data platforms: Databricks, Snowflake and cloud‐native data services Data architecture: Lakehouse, medallion, data mesh and domain‐oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise‐scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud‐native orchestration services Event streaming: Kafka and cloud‐native event‐streaming technologies Data modelling: Dimensional, analytical and domain‐oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

tooling, or cloud cost management across at least one major cloud provider (Azure, AWS, or GCP). Technical Skills Data engineering: strong proficiency with Apache Spark, dbt, Airflow/Azure Data Factory or equivalent orchestration, Data Lake/Apache Iceberg, and advanced SQL for large‐scale transformations. Microservices ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

skills: - Strong commercial full stack engineering capabilities - node, Typescript, Python, Java etc - Solid front end understanding - React, Next, Vue.js etc - Experience building scalable pipelines (Apache Spark, Airflow, dbt etc) - Experience building data lakes, data warehouses and/or real-time streaming solutions would be a real plus - Strong ...

Azure SQL Database, or Azure Data Factory Experience with Python or another object-oriented programming language Experience with Azure Databricks and/or Apache Spark Experience with reporting or visualisation tools such as Power BI, MicroStrategy, or similar platforms Experience with CI/CD workflows, including automated build, test ...

tooling, automated testing and reliable deployment processes. Startup Mindset: Proactive, resourceful and comfortable taking responsibility in a small, fast‐moving engineering team. Experience with Apache Airflow, Kubernetes, real‐time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial‐market systems would be useful ...

work effectively with technical teams including Data Scientists and MLOps Engineers. Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/vector ...