1 to 25 of 27 Permanent Apache Jobs in the South West

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Python, Java or Scala, and SQL. Hands-on experience building and operating large-scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. Experience working with cloud-native data platforms ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

used in modern AI systems Strong Python programming experience A habit of making, and justifying, architectural calls, not just executing them Nice to Have: Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, MongoDB, Qdrant/Milvus/pgvector, Pandas/Polars, cloud infra (GCP/AWS/Azure), and time ...

Essential Skills of the Analytical Platform Engineer Python Kubernetes Docker CI/CD pipelines (GitLab or similar) Desirable skills MLOps and data engineering concepts Apache Spark Scala S3-compatible object storage Graph database technologies If you are an driven and experienced Analytical platform engineer looking for their next exciting ...

configuration management and an interest in migrating to Ansible and cloud native tooling. Experience managing Kubernetes platforms and deploying containerised applications. Knowledge of GitLab, Apache NiFi, HashiCorp Vault, Trend Micro Deep Security Manager, Ivanti device control, and Ansible Tower/AWX. Windows Infrastructure Experience with Windows Server administration, deployment ...

event-driven technologies, such as:Testing & Automation: PyTest, Cucumber, Behave, DBT, GreatExpectations, GCP DVT, Monte Carlo, Soda, Deequ, RestAssured etcData Engineering & Orchestration: BigQuery, Spanner, Apache Kafka, Airflow, Spark, Cloud Composer, DAGs, Apache Beam, Pub/Sub, Dataflow, DataStage, Teradata, Snowflake, ETL, SQL etcData Governance & Visualisation: Looker, Ataccama, Dataplex ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition and design ...

with Red Hat OpenShift and Ceph storage platforms. Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark). Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

working with Red Hat OpenShift and Ceph storage platforms.Programming experience (e.g. Python, JavaScript)Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark).Experience delivering multi-system integration projects within defence, government, or highly regulated environments.Experience implementing and configuring training simulation technologies across ...

Splunk/ELK stackAbility to pro-actively manage own tasking and react to higher priority tasksExperience of Linux server administrationExperience in technologies such as Apache NiFi, MinIO/AWS S3Experience managing and patching Java and Python applicationsExperience with containerisation technologies such as Docker/Podman and deployment technologies such ...

skills Hands-on experience with Databricks for scalable data engineering, analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimization Experience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure … data pipeline deployment and monitoring using CI/CD tools and infrastructure-as-code practices Hands-on expertise with distributed data processing frameworks (e.g., Apache Spark, Hadoop) and real-time data streaming technologies (e.g., Kafka, Kinesis, Spark) Track record of collaborating with cross-functional teams (engineering, analytics, business stakeholders ...

hold active Enhanced DV Clearance (West) Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL Who Are We? Our client is a trusted and growing supplier to the National Security sector, delivering mission-critical solutions that … Engineer Should Have: Active eDV clearance (West) Willingness to work full time on site in Gloucester when required. Required technical experience in the following: Apache Kafka Apache NiFI SQL and noSQL databases (e.g. MongoDB) ETL processing languages such as Groovy, Python or Java To be Considered: Please either ...

hold active Enhanced DV Clearance (West) Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL Who Are We? Our client is a trusted and growing supplier to the National Security sector, delivering mission-critical solutions that … Engineer Should Have: Active eDV clearance (West) Willingness to work full-time on-site in Gloucester when required. Required technical experience in the following: Apache Kafka Apache NiFI SQL and noSQL databases (e.g. MongoDB) ETL processing languages such as Groovy, Python or Java To be Considered: Please either ...

hold active Enhanced DV Clearance (West) Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL Who Are We? Our client is a trusted and growing supplier to the National Security sector, delivering mission-critical solutions that ...

strong skills in several of the following Modern JavaScript GitLab CI/CD Observability Technologies Web API Integration GitLab Version Control Database Technologies OpenShift Apache NiFi Message Broker Technologies Java Web Auth Distributed Architecture Kubernetes SecDevOps Cloud Native Technologies The Interview Process We believe in keeping the interview process ...

field. Experience with LLM agents, autonomous multi-step reasoning systems, meta-learning, or self-improving ML pipelines. Experience with large-scale data pipelines (e.g., Apache Beam) or foundation model training and fine-tuning at scale. Experience working in research environments or track record of published research in relevant ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

ready data processing solutions using BigQuery, Pub/Sub, Dataflow and DataprocExperience developing end-to-end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam.Expert understanding of when to use a range of data storage technologies including relational/non-relational, document, row-based ...

lake. Translate architectural designs into implementation plans, work packages and practical technical delivery approaches. Lead API-based, file-based, legacy and bespoke integrations using Apache NiFi and AWS services. Manage the onboarding of source systems, including interface analysis, data mapping, transformation rules and security considerations. Define and coordinate unit … Significant experience leading enterprise-scale data engineering or integration programmes. Strong, hands-on AWS delivery experience, including secure engineering practices and operational observability. Strong Apache NiFi experience, including pipelines, processors, controller services, flow governance and operational monitoring. Proven experience delivering API integrations, file-based integrations, legacy system interfaces ...

cross‐functional teams and setting best practice standardsDesigning data models, ETL pipelines, warehousing solutions and distributed systemsBuilding and supporting real‐time streaming platforms using Apache Flink and related technologiesEnsuring compliance with industry standards and government regulationsTracking delivery against agreed designs and managing design deviationsSupporting pre‐sales and bids with … experience in data architecture on complex, secure programmesStrong hands‐on AWS experience (EC2, S3, RDS, Lambda, Redshift); Azure or GCP a bonusProven experience using Apache Flink for real‐time, high‐throughput data processingExperience leading, mentoring and developing technical teamsConfident engaging with senior technical and non‐technical stakeholdersKnowledge of recognised ...

teams and setting best practice standards Designing data models, ETL pipelines, warehousing solutions and distributed systems Building and supporting real-time streaming platforms using Apache Flink and related technologies Ensuring compliance with industry standards and government regulations Tracking delivery against agreed designs and managing design deviations Supporting pre-sales … data architecture on complex, secure programmes Strong hands-on AWS experience (EC2, S3, RDS, Lambda, Redshift); Azure or GCP a bonus Proven experience using Apache Flink for real-time, high-throughput data processing Experience leading, mentoring and developing technical teams Confident engaging with senior technical and non-technical stakeholders ...