2 of 2 Spark SQL Jobs in the City of London

skills/knowledge/experience: Experience of programming under Microsoft Azure Cloud Platform Experience of programming under Microsoft Fabric Platform Have knowledge of Spark Programming Ability to write Spark code for large scale data processing, including RDDs, Data Frames, and Spark SQL … Python/Notebook programming PySpark programming Spark Streaming/batch processing Delta table Optimization Fabric spark jobs Java programming language, OOP knowledge Database knowledge, including Relational Database and Non-SQL database. Experience of using the tools: Gitlab, Python unit test, CICD pipeline. Good skill ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy … throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need to program using the languages such ...