5 of 5 Apache Spark Jobs in Central London

within the Insurance industry Strong proficiency in programming languages such as Python, Java, or Scala. Experience with cloud platforms like Azure. Knowledge of big data technologies such as Hadoop, Spark, or Kafka. Proficiency in SQL and database management systems. Familiarity with data warehousing concepts and tools. Ability to work collaboratively with cross-functional teams. A solid understanding of data More ❯

data modelling techniques + data integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. Azure Data Engineer certification preferred More ❯

teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation Contribute to containerised deployments (Docker/… within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/Kubernetes) is a plus Excellent communication and stakeholder engagement skills Experience working within More ❯

communication skills. NICE TO HAVE: Hands-on experience with LLMs and Natural Language Processing (NLP) , including fine-tuning or prompt engineering. Familiarity with distributed computing or parallel processing (Ray, Spark, etc.). Experience deploying models in production environments (Docker, cloud services). Exposure to data engineering or working alongside data pipeline teams. A genuine passion for AI innovation and More ❯

logic Apply data validation, cleansing, and profiling techniques to ensure accuracy and consistency Implement access controls, data masking, and compliance-aligned security protocols Tune workloads and optimise performance across Spark, Fabric, and Azure components Translate business requirements into technical solutions through close collaboration with analysts and stakeholders Maintain clear documentation and contribute to internal knowledge repositories Essential Skills Strong … experience developing within Microsoft Azure and Microsoft Fabric: Proficiency in Spark programming including DataFrames, RDDs, and Spark SQL Python and PySpark development experience, including notebook-based workflows Hands-on experience with Spark streaming and batch processing Delta table optimisation and Fabric Spark job development Solid Java programming and OOP understanding Experience working with relational and NoSQL More ❯