4 of 4 Apache Spark Jobs in South London

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

plans and resource requirements accordingly. High proficiency in Python and SQL. Understanding of Snowflake's machine learning capabilities and integration with tools like Python, Spark, or DataRobot. Strong understanding of data governance, privacy, and security principles including IG and GDPR. Familiarity with version control tools (e.g. GitHub or Azure ...

improve reliability, maintainability, scalability and reuse.Your experience is likely to include several of the following: Microsoft Fabric Snowflake Azure OneLake Lakehouse architecture Data Factory Spark Python SQL ETL/ELT dimensional modelling Data Vault orchestration Git Azure DevOps/GitHub CI/CD Infrastructure as Code data quality MLOps … data they receive. Optimise Microsoft Fabric, Snowflake and Azure environments improve performance, scalability, resilience and resource utilisation across OneLake, Lakehouse, Data Factory, Spark/Notebooks, Snowflake and associated cloud services. Person Specification Qualifications Essential Educated to master's level or equivalent level of experience working at a senior level ...

Scala programming to develop robust, scalable, and high-performance data processing applications. Build and optimise large-scale data processing and analytics pipelines using Apache Spark. Develop and maintain end-to-end data pipelines for both batch and real-time data processing. Apply solid understanding of data engineering concepts, including … business stakeholders. Skills & Experience: Proven experience as a Software Engineer with a strong focus on Scala development. Extensive hands-on experience with OpenShift and Apache Spark. In-depth understanding of data engineering principles and practises. Experience in developing and maintaining scalable data pipelines for batch and real-time processing. ...