Prometheus, or Azure Monitor.• Maintain best practices for data engineering performance, testing, and documentation.Technical Skills Required• Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake.• Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies.• Experience tuning job configurations (e.g., memory management, parallelism, caching, adaptive … query execution).• Familiarity with orchestration tools: Databricks Workflows, Airflow, or Azure Data Factory.• Proficiency in Python or Scala for data engineering and pipeline development.• Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.• Knowledge of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade ...