76 to 94 of 94 Apache Spark Jobs in England

execute. You have deep experience designing, building, and operating production data pipelines, including owning reliability, monitoring, and failure recovery for orchestration systems such as Apache Airflow (or equivalent). You are fluent in modern data and cloud ecosystems, with hands-on experience (or the ability to ramp quickly … technologies such as Iceberg, Spark, PostgreSQL, and cloud-native infrastructure on AWS (or equivalent). You write clean, readable, and testable Python code, applying sound abstractions, naming, and review discipline to build systems that scale beyond the first version. You bring strong analytical and problem-solving skills ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive ...

optimise scalable data pipelines (batch and real-time) Work primarily with Java , with exposure to Python and streaming frameworks such as Kafka, Flink, or Spark Develop event-driven architectures and streaming integrations Integrate data from multiple sources using AWS services (e.g. Kinesis, Lambda, Glue, MSK) Architecture & Data Management Contribute … experience in Data Engineering or large-scale data systems Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark (streaming focus is a strong advantage) Solid understanding of streaming concepts (event time, state, backpressure) Experience building ETL/ELT or real-time data ...

learning or LLM-based systems Exposure to enterprise data platforms or similar tools Frontend development experience (e.g., React) Familiarity with distributed computing frameworks (e.g., Spark) What’s on Offer Competitive compensation package, including equity Comprehensive benefits Hybrid working model Opportunity to build products and systems from the ground ...

year; 1–3 years; 3–5 years; more than 5 years. Core technologies Akka (Actors, Streams, Persistence, Clustering, Projection, HTTP); ZIO; Play; Spark Libraries Circe; Cats; Cats Effect; Slick; Doobie; HTTP4S; Shapeless; Monix Databases RDBMS; MongoDB; Redis; Cassandra; Elastic Search Messaging Kafka; RabbitMQ Architecture and Cloud Microservices; DDD; Event ...

Transformation You will develop and improve UCLH's data warehouse environments. Being involved in the transition from using an R-based data pipeline to Spark jobs running on a data platform. You will model and transform data from our Epic electronic health record system, coordinating closely with stakeholders … clinical safety and data governance standards. Desirable Experience of using R or Python Experience with modern data engineering tools and frameworks (e.g. dbt, snowflake, spark, etc.) Experience in an NHS informatics role or other healthcare information environment Skills and Abilities Essential In depth specialist knowledge of information analysis ...

maintainable code Participating in code reviews, pairing, and team delivery Tech Stack & Environment Backend: Scala, Java (Python desirable) Frontend: HTML, CSS, JavaScript Frameworks: Play, Spark Data: Postgres, MongoDB, Hudi Cloud: AWS (S3, ECS, Aurora, Lambda) DevOps: Jenkins, Airflow, Terraform Monitoring: Kibana, Grafana Ways of working: Agile (Scrum/Kanban … Agile, multidisciplinary teams Strong analytical and problem-solving skills Clear communicator and collaborative team player Desirable Python, Docker, Airflow Big data tools (e.g. Spark, EMR) Experience with data platforms or risk/fraud systems How You'll Work Deliver value iteratively and early Collaborate closely across disciplines Write simple ...

handling edge cases, improving data quality and solving complex data challenges. ·The environment is modern and cloud based, with exposure to technologies such as Spark, Snowflake and BigQuery across AWS, GCP and Azure. ·You will also have the opportunity to support and mentor other engineers through collaboration, code reviews … Looking For ·Strong Python and SQL expertise gained within large scale production environments ·Experience delivering complex data transformation projects end to end ·Experience with Spark, Snowflake, BigQuery or similar technologies ·Exposure to cloud environments such as AWS, GCP or Azure ·Comfortable working with large, messy and evolving datasets ·Experience ...

A leading organisation undergoing major investment in its data and engineering capabilities is looking for a Senior Automation QA Engineer to strengthen quality across complex, distributed systems. This role sits at the intersection of software ...

A leading organisation undergoing major investment in its data and engineering capabilities is looking for a Senior Automation QA Engineer to strengthen quality across complex, distributed systems. This role sits at the intersection of software ...

systems, and modern cloud architectures . What youll be doing Design and build scalable data pipelines (batch & streaming) Work with technologies like Kafka, Flink, Spark, AWS services Develop real-time, event-driven data solutions Contribute to data architecture and modelling best practices Implement CI/CD pipelines and Infrastructure … deliver clean datasets What were looking for Strong programming skills in Java (preferred) or Python Experience with streaming technologies (Kafka/Flink/Spark) Knowledge of distributed systems and large-scale data processing Experience with cloud platforms (AWS preferred) Familiarity with DevOps tools and containerisation (Docker/Kubernetes ...

Developing and maintaining scalable data pipelines using Python - Automating manual Excel/VBA workflows into reliable, production-grade systems - Working with large datasets using Spark and Azure (Databricks/Synapse) - Supporting data modelling and analytics use cases (Crossing over to more Data Science driven work) - Driving best practices including … code reviews - Collaborating with stakeholders to deliver actionable data solutions What they’re looking for: - Strong Python and SQL skills - Experience or exposure to Spark, big data tools, or cloud platforms (ideally Azure) What’s on offer: - The chance to join a business at an exciting stage of growth ...

workloads. Practical experience working with AWS cloud services, including compute, storage, and security tooling. Infrastructure-as-code experience using Terraform. Distributed data processing using Apache Spark. Workflow scheduling and orchestration with Apache Airflow. Containerisation and environment management using Docker. CI/CD pipeline implementation and maintenance using GitLab. ...

Educated to degree level in a relevant IT subject. Expertise in designing and implementing data pipelines using Azure services (Azure Data Factory, Data storage), Spark and Databricks. Expertise in data modelling, database design and designing enterprise data architecture . Azure Data Stack SQL server, NoSQL, Nanobricks, ADT, Spark ...

Senior Developer (Python, Gen AI, Vector DB, Spark) Role Overview We are seeking a highly skilled Senior Developer with expertise in Python, Generative AI, Vector Databases, and Spark. The role focuses on building scalable AI-driven applications and data processing solutions. Python Development Develop robust applications and APIs using … performance and scalability. Vector Database Work with vector databases like Pinecone, FAISS, Weaviate. Develop embedding pipelines and semantic search. Optimize indexing and retrieval performance. Spark & Data Processing Build data pipelines using Apache Spark. Handle large-scale data processing. Optimize performance and scalability. CI/CD & DevOps Implement ...

teams and setting best practice standards Designing data models, ETL pipelines, warehousing solutions and distributed systems Building and supporting real-time streaming platforms using Apache Flink and related technologies Ensuring compliance with industry standards and government regulations Tracking delivery against agreed designs and managing design deviations Supporting pre-sales … data architecture on complex, secure programmes Strong hands-on AWS experience (EC2, S3, RDS, Lambda, Redshift); Azure or GCP a bonus Proven experience using Apache Flink for real-time, high-throughput data processing Experience leading, mentoring and developing technical teams Confident engaging with senior technical and non-technical stakeholders ...

Aker Systems was founded in 2017 by a team of experienced technology professionals who recognised an opportunity to provide highly secure enterprise data platforms to large organisations. We build and operate ground-breaking, ultra-secure ...

backend services using Java Work within cloud-native AWS environments Contribute to containerised workloads (Docker/Kubernetes) Build and support data-intensive systems (including Spark where applicable) Collaborate with architects and engineers on system design and delivery Participate in agile delivery, code reviews, and technical discussions Required Skills & Experience … background Experience working in distributed systems Conditional (must meet at least 2 of the below) AWS 3+ years hands-on experience Big Data/Spark 3+ years Containerisation Docker and/or Kubernetes, 3+ years Payments domain 23 years experience Nice to Have Infrastructure as Code (Terraform) Event-driven ...

Senior DataEngineer-RemoteWorking LynxarecurrentlyworkingwithalargeITconsultancytohelpthemsourceaDataEngineer.OurclientislargeglobalConsultancywhoworkwithenterpriseclients,solvingtheirbusinessandtechnologyproblemsusingcuttingedgesolutions. Asa DataEngineer ,youwillplayakeyroleindesigning,building,andmaintainingrobustdataplatformsandpipelines.Youllworkalongsidedatascientists,analysts,andengineerstoensurereliable,scalable,andhigh-qualitydatasolutions. Whatyoullbedoing: Designingandbuildingdatapipelines,models,andarchitectures DevelopingandmaintainingETL/ELTprocessesusingmoderntools Implementingdatastoragesolutionsacrossrelational,non-relational,andcloudplatforms Ensuringdataquality,reliability,andoperationalexcellence Leadingthedesignanddeliveryofnewdataproductsandpipelines Collaboratingcloselywithstakeholdersandtechnicalteams Providingtechnicalleadershipandmentoringwithindeliveryteams Whattherelookingfor: Strongexperienceindataengineeringroles Provenleadershiporlinemanagementexperience ...