1 to 25 of 68 Apache Spark Jobs in the UK

with Red Hat OpenShift and Ceph storage platforms. Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark). Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

trusted datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/ ...

Python, Java or Scala, and SQL. Hands-on experience building and operating large-scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. Experience working with cloud-native data ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

continuous learning to understand new data management systems Strong proficiency in hands-on development using industry-standard tools and languages (e.g., SQL, Python, Spark, cloud platforms), with a commitment to code quality and delivery excellence Preferred qualifications, capabilities, and skills Hands-on experience with Databricks for scalable data engineering … analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimization Experience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure, GCP), including migration from legacy systems Advanced proficiency in designing ...

engineers should be adept at Extract, Transform, Load (ETL) processes. This includes using frameworks such as the Medallion Architecture, as well as tools like Apache Airflow, Fivetran, Azure Data Factory, and Amazon Glue to automate data workflows and ensure data quality. Familiarity with cloud computing services, such … Cloud Platform, is increasingly important. Data engineers often leverage these platforms for scalable data storage and processing capabilities. Knowledge of big data frameworks like Apache Spark, Beam is crucial for handling large datasets and real-time data processing. Understanding data modelling techniques helps data engineers design efficient data ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

10+ years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure ...

/ELT processes and data pipeline development Experience working with cloud platforms such as AWS, Azure or GCP Knowledge of technologies such as Spark, Kafka, Airflow or similar Experience with databases, data modelling and data warehouse solutions Understanding of CI/CD, software engineering best practices and secure development ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

messaging architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture Hybrid cloud environments Enterprise integration patterns Modern data formats including Parquet, Avro, and ORC DevOps ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

twin platforms. Experience with vector databases, embeddings, semantic search, or RAG. Experience with streaming data platforms such as Kafka, Pub/Sub, Flink, or Spark Streaming. Experience supporting closed-loop automation, predictive assurance, or self-healing network use cases. Experience working with OSS systems, inventory platforms, assurance systems ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

understanding of Terraform enterprise. Experience in delivering system design, application development, testing, and operational stability. Knowledge of Big Data distributed compute frameworks like Spark, Glue, MapReduce etc. Excellent troubleshooting, analytical, and communication skills. Preferred Qualifications, Capabilities, and Skills: Multi Region disaster recovery setup, monitoring and testing. Experience in Data ...

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

platform and lakehouse transformation journeys What We're Looking For Strong commercial experience in Data Engineering Hands-on experience with Databricks and/or Apache Spark Strong PySpark and SQL development skills Experience designing, building and supporting data pipelines Experience working with Azure or AWS Good understanding ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...