7 of 7 Remote Contract Apache Spark Jobs

contract assignment. Key Requirements: Proven background in AI and data development Strong proficiency in Python , including data-focused libraries such as Pandas, NumPy, and PySpark Hands-on experience with Apache Spark (PySpark preferred) Solid understanding of data management and processing pipelines Experience in algorithm development and graph data structures is advantageous Active SC Clearance is mandatory Role Overview … You will play a key role in developing and delivering advanced AI solutions for a Government client . Responsibilities include: Designing, building, and maintaining data processing pipelines using Apache Spark Implementing ETL/ELT workflows for large-scale data sets Developing and optimising Python-based data ingestion tools Collaborating on the design and deployment of machine learning models … performance across distributed systems Contributing to data architecture and storage strategy design Working with cloud data platforms (AWS, Azure, or GCP) to deploy scalable solutions Monitoring, troubleshooting, and tuning Spark jobs for performance and cost efficiency Engaging regularly with customers and internal stakeholders This is an excellent opportunity to join a high-profile organisation on a long-term contract More ❯

cloud data engineering, with a strong focus on building scalable data pipelines Expertise in Azure Databricks, including building and managing ETL pipelines using PySpark or Scala Solid understanding of Apache Spark, Delta Lake, and distributed data processing concepts Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and Azure Synapse Analytics Proficiency in SQL and Python More ❯

data modelling techniques + data integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. Azure Data Engineer certification preferred More ❯

data modelling techniques + data integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. Azure Data Engineer certification preferred More ❯

experience with AWS data platforms and related services. Solid grasp of data governance principles, including data quality, metadata management, and access control. Familiarity with big data technologies such as Spark and Hadoop, and distributed computing concepts. Proficiency in SQL and at least one programming language (e.g., Python, Java). Preferred Qualifications: Relevant certifications in data architecture, cloud platforms, or More ❯

experience with AWS data platforms and related services. Solid grasp of data governance principles, including data quality, metadata management, and access control. Familiarity with big data technologies such as Spark and Hadoop, and distributed computing concepts. Proficiency in SQL and at least one programming language (e.g., Python, Java). Preferred Qualifications: Relevant certifications in data architecture, cloud platforms, or More ❯

in developing application using SpringBoot Experience in developing web infrastructure (Solr, kubernetes) Experience in git and basic Unix Commands You may also have Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. Our team work at least two days More ❯