1 to 25 of 47 Apache Jobs in the UK excluding London

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. Power BI semantic modelling. APIs and cloud business system integration. Data governance, lineage and quality frameworks. Professional services experience. What ...

highly desirable but strong Python/PySpark engineers who can learn Foundry quickly are encouraged to apply. Extensive experience with Python and PySpark/Apache Spark. Proven track record designing and delivering scalable data pipelines and data products. Strong SQL and data modelling skills. Experience with modern data architectures ...

architecture standards and best practices Working closely with business and technical stakeholders Leading data transformation and cloud migration initiatives Desirable Databricks, Snowflake, dbt Kafka, Apache Iceberg, Delta Lake Data Mesh/Data Fabric AI & ML-enabled data platforms ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance practices ...

Skills & Experience Strong commercial experience in data engineering within cloud-based environments. Advanced Python and SQL development skills. Hands-on experience with Azure Databricks, Apache Spark and PySpark. Good understanding of Delta Lake, lakehouse architecture and modern data modelling. Experience building and managing production data pipelines, including batch ...

with monitoring/logging tools (ELK stack, Prometheus, Splunk) Familiarity with service mesh and modern networking (Istio, Envoy) Exposure to data engineering and integration (Apache ecosystem, SQL, Python) Recognised technical certifications (AWS/Azure Architect, Kubernetes CKA/CKAD, Terraform) Experience in influencing technical standards, best practices, or contributing ...

containerisation and orchestration technologies including Kubernetes, Docker, ECS, EKS, and AKS . Experience working with distributed event-streaming technologies such as Kafka and Apache Flink . Strong troubleshooting skills with the ability to resolve complex technical and operational issues. Proven ability to lead, mentor, and develop engineers within Agile ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise • Observability – Experience ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

Proficiency in data quality assurance, data governance, and query optimization. Nice to have: Familiarity with the AWS data ecosystem (S3, Glue, EMR/PySpark, Apache Iceberg). Non-Technical Skills & Competencies Proven sense of ownership, commitment, and personal accountability. Meticulous attention to detail with a proactive problem-solving mindset. ...

Familiarity with monitoring tools such as CloudWatch, Prometheus or Grafana. Useful additional experience Relevant AWS certifications. Exposure to data-processing environments using Databricks or Apache Spark. The person we are looking for You enjoy making systems easier to run and reducing the need for manual intervention. You investigate problems ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable."Nice to Have"Azure, Databricks, Apache Spark or comparable cloud and data platforms. Data pipelines, ETL and workflow-orchestration technologies. Machine-learning methods such as forecasting, prediction or classification. Experience ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will be joining … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure data ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will be joining … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure data ...

Snowpark Iceberg Tables Secure Data Sharing Snowflake Cortex Data Integration Fivetran SnapLogic Enterprise SaaS integration patterns Data Transformation & Orchestration dbt Cloud/dbt Core Apache Airflow (MWAA) CI/CD pipelines Event-driven orchestration Data Governance & Observability Collibra Data Catalogue Data Lineage Data Governance Frameworks Acceldata Data Observability Data ...

build elegant solutions Understanding of system administration tasks on some of the following; Operating systems (e.g. Linux, CoreOS and Windows) Application stacks (e.g. nginx, Apache, Tomcat, JBoss, Websphere, Hybris, IIS, .Net, Node.js, Solr) RDBMS (e.g. PostgreSQL, MySQL, Oracle, MS SQL) NoSQL (e.g. MongoDB, Cassandra, Riak) Containerisation and microservices (e.g. ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

role). Cloud Infrastructure & Data Pipelines: Build with AWS and Azure services to construct ETL workflows and dynamic systems using orchestration platforms like Apache Airflow. Production Operations: Support production applications, maintain SLAs, and create continuous evaluation metrics for ML solution performance. Qualifications & Skills 2-5 years' hands-on software ...

knowledge sharing and mentoring across the team. What we are looking for Practical experience managing Databricks in a production environment. A strong understanding of Apache Spark, Delta Lake and Unity Catalog. Experience working with one or more major cloud providers, including storage, networking and identity management. Confidence using Terraform ...

effectively with stakeholders and champion better ways of working. Nice to Have Experience with Docker, Java/Scala, Kafka, Pulsar, Azure Event Hub, Collibra, Apache Iceberg, Apache Arrow, PySpark or AI engineering technologies such as Snowflake AI, LangChain, LlamaIndex or Pinecone would be advantageous. This is a great ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated desk ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

explore new technologies, share knowledge, and contribute to a culture of excellence and innovation. Design & Build: Develop and optimise data pipelines using Databricks, Apache Spark, and Python. Develop & Deliver: Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services. Integrate & Automate: Implement ETL processes and data … Hands-on commercial expertise with Databricks. You should have: Proven experience as a Data Engineer working with large, complex datasets. Hands-on expertise with Apache Spark, and SQL. Strong proficiency in Python (PySpark preferred). Experience with AWS cloud services including S3, Glue, Lambda, IAM. Familiarity with ETL design ...