4 of 4 Permanent Apache Spark Jobs in Scotland

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. ...

comprehensive data analysis and reporting. Implement and uphold best practices in data governance, security, and compliance, using tools like Azure Purview, Unity Catalog, and Apache Atlas to maintain data integrity and trust. Ensure data quality and integrity through meticulous attention to detail and rigorous QA processes, continually refining … ease. Understanding of Data Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using ...

working on complex data pipelines at scale, you'll thrive here. Tech stack: Micro-services Container Platforms (Kubernetes, CRC, Docker) Big Data Technologies (Apache.... Spark, Flink, HDFS, Airflow, Trino, Iceberg, Hive) Dependency injection frameworks (Spring) Observability (Loki/Grafana) Large scale data processing (Kafka) CI/CD Build tools ...

analytics tools Familiarity with industry common BI and ETL solutions Hands-on experience in a technical role leveraging SQL, Python, Java and/or Spark to build, operate and maintain data analytics solutions Extensive knowledge of and experience with large-scale database technology (e.g. Snowflake, Netezza, Exadata, Teradata, Greenplum ...