2 of 2 Batch Processing Jobs in Derbyshire

our capabilities, solving new data problems and challenges every day. Key Responsibilities: Design, Build, and Optimise Real-Time Data Pipelines: Develop and maintain robust and scalable stream and micro-batch data pipelines using Databricks, Spark (PySpark/SQL), and Delta Live Tables. Implement Change Data Capture (CDC): Implement efficient CDC mechanisms to capture and process data changes from various … Databricks Lakehouse Platform is essential. Strong Spark and Python/SQL Skills: Proficiency in Spark programming (PySpark and/or Scala) and expert-level SQL skills. Real-Time Data Processing: Demonstrable experience in building and managing stream and micro-batch processing pipelines using technologies like Spark Structured Streaming or Delta Live Tables. Deep Understanding of Delta Lake More ❯

EC2/Node (ReactJS) using AWS API Gateway and Lambda Functions (Python) for the internal web application. The individual will be able to demonstrate a strong background in backend batch processing (ETL) using Python and SQL skills. Whilst having good knowledge developing web applications using ReactJS/Python. Our technology team uses the following tools for development GitLab More ❯