1 to 25 of 135 Apache Spark Jobs in the UK

financial services organisation Your new role We are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … performance, reliability, and scalability.The role is focused on building nonperformance Apache Spark jobs, with a strong emphasis on performance optimisation. It is primarily Java-based, with Python strongly encouraged; experience across Java, Python, and Scala is ideal, with Java and Python being the preferred combination.The role sits within ...

financial services organisation Your new role We are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … Engineer. You will be responsible for development work, sprint delivery, demos, documentation, and stakeholder engagement. This position suits a mid-level engineer with strong Spark development experience rather than design, infrastructure, or management responsibilities. What you'll need to succeed Strong hands-on experience with Apache Spark ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. What … which Azure data focused components to use for which purpose. Microsoft certified architect Databricks Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Ability to Design, Build and deploy ...

technologies, share knowledge, and contribute to a culture of excellence and innovation. Key responsibilities include: • Design & Build: Develop and optimise data pipelines using Databricks, Apache Spark, and Python. • Develop & Deliver: Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services. • Integrate & Automate: Implement ETL processes … Hands-on commercial expertise with Databricks. You should have: • Proven experience as a Data Engineer working with large, complex datasets. • Hands-on expertise with Apache Spark, and SQL. • Strong proficiency in Python (PySpark preferred). • Experience with AWS cloud services including S3, Glue, Lambda, IAM. • Familiarity with ...

help other members of the team. · Experience provisioning and managing AWS services using infrastructure as code such as AWS CloudFormation or Terraform. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Knowledge of Serverless technologies, framework and best practices. · Experience provisioning … managing Apache Airflow. · Exposure to using AI tools to enhance productivity and quality. Why Join RealityMine? At RealityMine, we believe our people are at the heart of everything we do. That’s why we go the extra mile to support every team member to unlock their full potential. Whether ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

NoSQL technologies Hands-on experience with at least one major cloud platform (Azure, AWS, or GCP) Experience with big data technologies such as Spark, Hadoop, or Kafka Proficiency with ETL/ELT tools and frameworks (e.g. Airflow, dbt, Informatica) Strong programming skills in Python, Java, Scala, or similar Experience ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

Eligibility for UK SC clearance Nice to have (but not essential) Exposure to AWS, Azure or GCP Familiarity with tools such as Airflow, dbt, Spark Awareness of CI/CD pipelines or containerisation Experience in public sector or regulated environments Important note for applicants This role is deliberately positioned ...

e.g., Kafka), and cloud services (e.g., AWS, GCP, Azure). Programming Skills: Any programming skills in languages like Python (NumPy, pandas, PySpark), Java (Core Spark with Java, functional interface, collections), or Scala with experience in automation and scripting would be great. Infrastructure Tools: Experience with containerization and orchestration tools ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

Time series, NLP, recommendation systems (as applicable) Programming Python (NumPy, Pandas, Scikit learn) Optional: R, SQL Data Platforms Relational & NoSQL databases Big data frameworks (Spark, Hive, Databricks) MLOps & Deployment Model lifecycle management CI/CD pipelines Containerization (Docker, Kubernetes desirable) Model packaging and REST APIs Cloud & Tools (Any combination ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. … version control. Data Engineering Stack Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

platforms (AWS, GCP, or Azure) Experience with data warehousing and lake architectures ETL/ELT pipeline development SQL and NoSQL databases Distributed computing frameworks (Spark, Kinesis etc) Software development best practices including CI/CD, TDD and version control Strong understanding of data modelling and system architecture Excellent problem ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

Confident communication skills, able to convey technical concepts to non-technical stakeholders. Bonus Points For Experience with real-time streaming tools like Kafka or Spark Streaming. Exposure to MLOps practices and maintaining ML models in production. Familiarity with data visualisation tools such as Power BI, Qlik, or Tableau. Experience ...

automation & configuration management; • Ansible (plus Puppet, Saltstack), Terraform, CloudFormation; • NodeJS, REACT/MaterialUI (plus Angular), Python, JavaScript; • Big data processing and analysis, e.g. Apache Hadoop (CDH), Apache Spark; • RedHat Enterprise Linux, CentOS, Debian or Ubuntu. • Java 8, Spring framework (preferably Spring boot), AMQP - RabbitMQ, • Open source technologies ...

automation & configuration management; • Ansible (plus Puppet, Saltstack), Terraform, CloudFormation; • NodeJS, REACT/MaterialUI (plus Angular), Python, JavaScript; • Big data processing and analysis, e.g. Apache Hadoop (CDH), Apache Spark; • RedHat Enterprise Linux, CentOS, Debian or Ubuntu. • Java 8, Spring framework (preferably Spring boot), AMQP - RabbitMQ, • Open source technologies ...

Data Factory, ADLS Gen2, Synapse, and Azure SQL. Ensure data solutions adhere to performance, scalability, security, and governance best practices. Guide the team on Spark performance tuning, cost optimization, and efficient cluster management. Implement Delta Lake, data modeling, and data quality frameworks. Stakeholder Management & Communication Act as the primary … deployment approaches to ensure high quality. Support release planning, sprint ceremonies, and delivery governance. Your Skills: Strong hands‐on experience with Azure Databricks and Apache Spark (PySpark/Scala). Solid experience with Azure services: Azure Data Factory Azure Data Lake Storage (Gen2) Azure Synapse Analytics Experience with ...