51 to 75 of 95 Apache Jobs in the UK

bandwidth. The goal: no software rewrites, predictable time to production, early integration, and fast iteration. You'll work with tools such as DBT, Python, Apache Airflow, Spark, Iceberg; AWS services like Lambda, Bedrock, and SageMaker, containers, and whatever adds the most value. You'll design tools that are simple ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated desk ...

modern AI systems. Strong programming experience in Python. Data & Infrastructure Experience with some of the following technologies is highly desirable: Frameworks/Infrastructure Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Libraries Pandas Polars Engineering Skills ...

understanding of the role of testing, DevOps etc. in the development cycle. An understanding of distributed messaging technologies, ideally including experience of JMS, Apache Pulsar or SolaceExperience with relational databases, such as MS SQL Server. Knowledge of frameworks such as Spring, Spring Boot, and Project Reactor/Spring Webflux ...

concisely. Good to HaveKnowledge of the adTech space and particularly the identifiers used in the field. Experience with large-scale distributed data processing (e.g., Apache Spark) and cloud data infrastructure across GCP and AWS (e.g., Dataproc, Dataflow, GKE, EMR).Experience with TemporalWe use automated decision systems (ADS) as part ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

explore new technologies, share knowledge, and contribute to a culture of excellence and innovation. Design & Build: Develop and optimise data pipelines using Databricks, Apache Spark, and Python. Develop & Deliver: Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services. Integrate & Automate: Implement ETL processes and data … Hands-on commercial expertise with Databricks. You should have: Proven experience as a Data Engineer working with large, complex datasets. Hands-on expertise with Apache Spark, and SQL. Strong proficiency in Python (PySpark preferred). Experience with AWS cloud services including S3, Glue, Lambda, IAM. Familiarity with ETL design ...

hold active Enhanced DV Clearance (West) Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL Who Are We? Our client is a trusted and growing supplier to the National Security sector, delivering mission-critical solutions that … Engineer Should Have: Active eDV clearance (West) Willingness to work full time on site in Gloucester when required. Required technical experience in the following: Apache Kafka Apache NiFI SQL and noSQL databases (e.g. MongoDB) ETL processing languages such as Groovy, Python or Java To be Considered: Please either ...

hold active Enhanced DV Clearance (West) Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL Who Are We? Our client is a trusted and growing supplier to the National Security sector, delivering mission-critical solutions that ...

cloud data AWS Aurora PostgreSQL AWS Database Migration Service (DMS) Snowflake Azure Google Cloud Platform (GCP) Data integration and engineering Informatica Talend Apache Airflow Python for data processing and automation Data warehouse design and optimisation APIs and event-driven architecture REST APIs GraphQL Apache Kafka Event-driven architectures ...

secure, scalable data solutions across the full lifecycle, including acquisition, integration, storage, processing and analysis. Develop and maintain data ingestion and transformation pipelines using Apache NiFi, Python and SQL. Design data models, schemas and data contracts/Interface Control Documents (ICDs) to support reliable integration between systems. Integrate data … take ownership of customer-facing deliverables. Strong Python and SQL skills, with experience building, testing and optimising data pipelines. Hands-on experience with Apache NiFi for data ingestion, routing, transformation and orchestration. Experience of data modelling and schema design, including defining data contracts or Interface Control Documents (ICDs). ...

that are unstructured or semi-structured. Proficiency in statistics and exploratory data analysis methods. Familiarity with data processing frameworks such as Spark, Ray, or Apache Beam. Ability to communicate technical findings clearly to research and product teams. Software Engineering IC4 - The typical base pay range for this role across ...

other genome browser Experience working in a multi-person team delivering production services Experience with large-scale biological datasets Experience working with the Apache Parquet file format Experience implementing AI/ML approaches for quality control and/or genome annotation Behaviours we value in our team: You bring ...

business. Key Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platformDevelop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta LakeContribute to platform architecture, engineering standards and best practices across the data estateWork with Product, Analytics and Data Science teams … contribute to continuous improvement across the engineering teamJob Requirements: Significant experience building and supporting scalable, production-grade data platformsStrong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQLGood understanding of cloud-based data engineering, ideally within Microsoft Azure environmentsExperience with Infrastructure as Code tools such ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python, SQL Essential ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python, SQL Essential ...

hybrid work from home policy. Qualifications Commercial experience with web client or server-side development Proficiency with Golang, TypeScript, React, SQL, Kafka or Apache Flink Background delivering software in continuous change environments Familiarity with AI tools to enhance developer productivity Clear technical communication with cross-functional stakeholders Proven grasp ...

Java and related technologies. Document code and processes to support knowledge sharing within the team. Profile Must haves: 2+ years industry experience Java Linux Apache Mongo/Postgres AI - prompt engineering or AI-assisted development (Claude, Cursor etc) Worked on a broker platform or similar industry experience a huge ...

something new, this is it. What you'll be doing Designing and building Real Time streaming pipelines on Kafka Architecting a lakehouse platform using Apache Iceberg Building high-performance analytics over large-scale and temporal (time-series) data Developing knowledge graphs, semantic models and ontologies to give the data ...

specific area or have broad knowledge across the systems engineering lifecycle, your skills matter. Your Impact Protecting RAF aircraft like Typhoon and Apache Advancing next-gen air combat with Team Tempest and GCAP Developing future electronic warfare systems Harnessing AI/ML to unlock intelligence from vast data What ...

data exchange platform. This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java. The platform will use open-source, cloud-agnostic technologies to ingest, transform, validate and exchange data securely between … systems. Skills and Experience: Proven experience designing and delivering data integration or ETL solutions using Apache NiFi. Practical project experience with RDF triples, SPARQL and semantic data modelling. Hands-on development experience in both Python and Java. Experience designing reusable pipelines across APIs, databases, files and other data sources. ...

Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data modelling and lakehouse design (bronze, silver, gold) Terraform, CI/CD and pipeline orchestration A track record of setting … pairing and design sessions Confidence communicating with technical and non-technical stakeholders, and challenging constructively Desirable: SQL Server or on-prem to cloud migration, Apache Flink, regulated industry experience If you'd rather shape a platform than inherit a finished one, and want a seat at the table while ...

activities. You will work closely with architects, senior engineers, and analysts to deliver scalable and reliable migration solutions, using technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML configurations. You should be comfortable working in a collaborative, delivery-focused environment and have a strong interest … repointing and migration to cloud environmentsData Pipeline Testing & Validation (Core Focus)Test ETL/ELT data pipelines on AWS services, including AWS Glue and Apache IcebergSupport validation of data pipeline migrations to AWS Data Lakehouse architecturesTest pipelines using: Python/PySpark transformationsSQL-based validation logicYAML-driven configurationsExecute and validate ...

maintaining new Local and Virtual Linux Server Environments Coding and maintenance of internal and external websites utilising standard LAMP stack or IIS, Apache, PHP, HTML, CSS, WordPress, Node Maintenance of MySQL and PostgreSQL databases both locally and remotely Helping staff with Windows 11 and application related enquiries Understanding network ...