2 of 2 Spark SQL Jobs in the North East

modeling, ETL/ELT development, and collaborative engineering practices. Key Responsibilities * Design, develop, and maintain scalable data pipelines using Azure Databricks (Python, PySpark, SQL). * Build and optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including … Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with data architects, analysts, and business stakeholders to understand data requirements. * Optimize Databricks clusters, jobs, and runtimes for performance and cost ...

comfortable operating in Agile teams, designing scalable solutions and working with varied datasets and architectures. Essential Qualifications Proficiency in scripting languages (Python, PowerShell, SQL, Scala or Spark-SQL) Experience with cloud platforms (AWS, Azure or GCP) Hands-on expertise with Databricks, Spark ...