4 of 4 Spark SQL Jobs in the North of England

modeling, ETL/ELT development, and collaborative engineering practices. Key Responsibilities * Design, develop, and maintain scalable data pipelines using Azure Databricks (Python, PySpark, SQL). * Build and optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including … Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with data architects, analysts, and business stakeholders to understand data requirements. * Optimize Databricks clusters, jobs, and runtimes for performance and cost ...

quality data solutions. Key Responsibilities: Develop, optimise, and maintain ETL/ELT pipelines within Databricks Build reliable data ingestion frameworks using PySpark and Spark SQL Design well-structured data models across medallion/lakehouse architecture Work with DevOps teams to automate deployments using CI/ ...

house team, this role could be the next step for you. Data Engineer Responsibilities: Design, build, and optimise data pipelines in Python, PySpark, SparkSQL, and Databricks. Ingest, transform, and enrich structured, semi structured, and unstructured data. Operate and support production grade data systems with strong observability and monitoring. Enable real ...

metadata-driven data pipelines using ADF and Databricks. Build and implement end-to-end metadata frameworks, ensuring scalability and reusability. Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing. Collaborate with cross-functional teams to integrate data solutions into enterprise architecture. Implement CI/CD pipelines … Expertise in building and orchestrating data pipelines. Databricks: Hands-on experience with notebooks, clusters, and job scheduling. Pandas: Advanced data manipulation and transformation skills. SparkSQL: Strong knowledge of distributed data processing and query optimization. CI/CD: Experience with tools like Azure DevOps, Git, or similar for automated deployments. Metadata ...