1 to 1 of 1 Apache Airflow Jobs in West Yorkshire

one of the biggest financial services organisations in the world. The successful candidate will be responsible for designing, building, and maintaining data pipelines using Apache Spark and Scala. This includes tasks like: Extracting data from various sources (databases, APIs, files) Transforming and cleaning the data Loading the data into … data warehouses or data lakes (e.g., BigQuery, Amazon Redshift) Automating the data pipeline execution using scheduling tools (e.g., Airflow) Work with Big Data technologies: You'll likely work with various Big Data technologies alongside Spark, including: Hadoop Distributed File System (HDFS) for storing large datasets Apache Kafka for … real-time data streaming Apache Hive for data warehousing on top of HDFS Cloud platforms like AWS, Azure, or GCP for deploying and managing your data pipelines Key skills: At least 8+ Years of IT Experience with designing, building, and maintaining data pipelines. At least 4+ Years of experience more »