2 of 2 Remote/Hybrid Apache Spark Jobs in Scotland

validation (e.g., schema tests, data quality checks) Manage data documentation and lineage within DBT Design and develop scalable data pipelines using Databricks and Apache Spark Build and optimize data workflows for batch and real-time processing Transform and process large datasets using PySpark, SQL, and Delta Lake Implement … terraform enterprise , Snowflake terraform provider knowledge. Snowflake : Snow sight , Python SDK for snowflake , Stage , storage integration , Data marketplace , direct shares , listings , catalog linked database , Apache Iceberg , external tables , Catalog integrations , Snowpipe , external volume , Cortex ai ,privatelink. Airflow : Dag creation , operators , connections , deployment on Kubernetes , Astronomer. We are a Disability ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...