23 of 23 Apache Spark Jobs in the North of England

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

Know Salary: Up to £80,000 per annum. Team: Work with cross‐functional teams and mentor junior engineers. Tools: Microsoft Fabric, Azure ML, MLflow, Spark, Python, SQL, CI/CD, and ML frameworks like TensorFlow, PyTorch, Scikit‐learn. Scope: End‐to‐end AI/ML solutions for multi‐product …/ML. Familiarity with Microsoft Fabric (Data Engineering, Data Science, Real‐Time Analytics) and Azure ML, including MLflow. Proficiency with tools such as Apache Spark, Azure Data Factory, GIT, Python and SQL. Skilled in developing, training and deploying machine learning models with frameworks like TensorFlow, PyTorch or Scikit ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...

driven decision making. What you'll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...

data engineering, ETL development or data platform delivery. Strong programming skills in Java or Python. Exposure to modern data technologies such as Kafka, Spark or Flink. Experience working within cloud environments, ideally AWS. Understanding of software engineering principles, version control and CI/CD practices. Familiarity with container technologies ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

Scala. Strong knowledge of SQL and experience with data warehousing solutions. Familiarity with cloud platforms (AWS, Azure, Google Cloud) and big data technologies (Hadoop, Spark). Excellent problem-solving skills and a keen eye for detail. A collaborative spirit and a genuine passion for data! Our Client's Benefits ...

Strong commercial experience in data engineering and modern data platforms. Advanced programming skills in Java or Python. Expertise working with technologies such as Kafka, Spark, Flink or similar distributed data frameworks. Experience delivering solutions within AWS or other major cloud platforms. Strong understanding of data architecture, scalability and performance ...

help deliver robust, secure and high-performing data products. What youll be doing: Design and build scalable data pipelines using Python, SQL, Spark and Databricks. Develop, deploy and optimise machine learning models, including NLP and generative AI solutions. Build data ingestion pipelines across batch and streaming environments. Design data … visualisations using Power BI or Tableau. Mentor junior team members and promote engineering best practices. What you need: Commercial experience with Python, SQL, Spark and modern data engineering frameworks. Experience deploying machine learning models into production. Strong knowledge of cloud platforms such as Azure, AWS or GCP. Experience with ...

Months contract 2 Days a month onsite - Expenses to Newcastle depends on their hub. Start 3 from offer Experience required AWS Services Terraform Apache Spark Python Airflow Docker Nice to haves Gitlab Security Scanning Agents (eg Trivy, Trend Micro, Wiz etc) Jupyter Notebooks Integration with DWP Integration Team ...

lead embedded data science teams, partnering effectively with senior stakeholders across multiple domains Hands‐on technical expertise with tools such as Databricks, Python, Spark, and GCP/BigQuery Engineering mindset, with experience moving teams toward machine learning engineering best practice Credibility to lead long‐tenured individual contributors while providing ...

scale. The position sits in a team delivering high-impact ML across marketing and advertising, working on both real-time and batch workloads with Spark, Python, Java, and #J-18808-Ljbffr ...

from ingestion and streaming through to warehousing, serving, and cost optimisation You have deep experience with streaming architectures – Kinesis, Kafka/MSK, Flink, or Spark Streaming – and know when each is the right choice You can model the cost implications of architecture decisions and advise clients on trade‐offs … between throughput, latency, durability, and spend AWS certification at professional or specialty level is strongly preferred Mastery of Python, SQL, and Spark – and strong opinions on how they should be used in production data pipelines Deep experience with dbt, Airflow, or equivalent orchestration – including how to build pipelines that ...

from ingestion and streaming through to warehousing, serving, and cost optimisation You have deep experience with streaming architectures – Kinesis, Kafka/MSK, Flink, or Spark Streaming – and know when each is the right choice You can model the cost implications of architecture decisions and advise clients on trade‐offs … between throughput, latency, durability, and spend AWS certification at professional or specialty level is strongly preferred Mastery of Python, SQL, and Spark – and strong opinions on how they should be used in production data pipelines Deep experience with dbt, Airflow, or equivalent orchestration – including how to build pipelines that ...

must have demonstrable experience designing, deploying, and operating big data platforms at scale. You will have hands-on experience with OSS data ecosystems (e.g., Spark, Trino, Airflow, Apache Superset, and similar technologies). You will have knowledge of cloud platforms and traditional on-premise hosting; hands-on experience ...

business use cases and/or to improve product/system performance, quality and accuracy. Near Real-Time and Batch Inferencing: Use infrastructure like Spark and Ray to stand up inferencing services that integrate with operational/analytics workloads. ML Infrastructure: Help build a first-class machine learning platform … learning on real recommendations use cases (brownie points for productionised sequential learning use cases!). Experience with ML/distributed ML frameworks like Ray, Spark-MLlib, TensorFlow etc. Experience with real-time scoring/evaluation of models with low latency constraints. Great coding skills and strong software development experience ...

business use cases and/or to improve product/system performance, quality and accuracy. Near Real‐Time and Batch Inferencing: Use infrastructure like Spark and Ray to stand up inferencing services that integrate with operational/analytics workloads. ML Infrastructure: Help build a first‐class machine learning platform … learning on real recommendations use cases (brownie points for productionised sequential learning use cases). Experience with ML/distributed ML frameworks like Ray, Spark‐MLlib, TensorFlow etc. Experience with real‐time scoring/evaluation of models with low latency constraints. Great coding skills and strong software development experience ...