51 to 73 of 73 Delta Lake Jobs

cloud and AI stacks such as Azure OpenAI/OpenAI, Azure ML, Databricks, Kubernetes, and Docker. Familiarity with data platforms such as Azure Data Lake, Synapse/ADF, Delta Lake, and messaging platforms (e.g., event hubs, service bus). Exposure to MLOps and observability tools (e.g., MLflow ...

scalable, secure, and high-performance data solutions across the organisation. Role and Responsibilities: Lead design and implementation of enterprise data architecture (Azure, Databricks, Data Lake, Data Factory). Enable AI/ML workflows and MLOps integration. Define engineering standards, governance frameworks, and architectural blueprints. Ensure compliance with GDPR … analysts in best practices. To apply you should have the following skills & experience: Proven experience architecting data platforms on Microsoft Azure (Databricks, Data Lake Storage Gen2, Azure Data Factory). Strong knowledge of Lakehouse architecture, Delta Lake, PySpark, SQL. Experience with AI/ML workflows, data governance ...

production workloads on Azure + Databricks . Strong PySpark/Spark SQL and distributed data processing expertise. Proven Medallion/Lakehouse delivery experience using Delta Lake . Solid dimensional modelling (Kimball) including surrogate keys, SCD types 1/2, and merge strategies. Operational experience-SLAs, observability, idempotent pipelines … watermarking) Subscription/entitlement modelling experience Advanced Unity Catalog security (RLS, ABAC, PII governance) Terraform/Bicep for IaC Fabric Semantic Model/Direct Lake optimisation ...

data engineering, data platforms and big data technologies Evaluate and improved existing data processes Essential Experiences: Advanced SQL skills Proficient in Databricks, including Spark, Delta Lake and MLflow Experience migrating SSIS to Databricks In depth understanding of ETL/ELT processes Knowledge of data warehousing concepts and architectures ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

that directly underpin operational decision-making and analytics for live energy assets. Key responsibilities Design, build, and maintain scalable data pipelines using Databricks (including Delta Live Tables). Develop robust ETL/ELT workflows ingesting data from operational, telemetry, and third-party systems. Optimise pipeline performance, reliability, and cost … from these platforms. Technical requirements Proven experience as a Data Engineer in production environments. Strong Python and SQL skills. Hands-on Databricks experience (DLT, Delta Lake; Unity Catalog desirable). Solid understanding of data modelling, data warehousing, and distributed systems. Experience with cloud data platforms (Azure preferred ...

that directly underpin operational decision-making and analytics for live energy assets. Key responsibilities Design, build, and maintain scalable data pipelines using Databricks (including Delta Live Tables). Develop robust ETL/ELT workflows ingesting data from operational, telemetry, and third-party systems. Optimise pipeline performance, reliability, and cost … from these platforms. Technical requirements Proven experience as a Data Engineer in production environments. Strong Python and SQL skills. Hands-on Databricks experience (DLT, Delta Lake; Unity Catalog desirable). Solid understanding of data modelling, data warehousing, and distributed systems. Experience with cloud data platforms (Azure preferred ...

that directly underpin operational decision-making and analytics for live energy assets. Key responsibilities Design, build, and maintain scalable data pipelines using Databricks (including Delta Live Tables). Develop robust ETL/ELT workflows ingesting data from operational, telemetry, and third-party systems. Optimise pipeline performance, reliability, and cost … from these platforms. Technical requirements Proven experience as a Data Engineer in production environments. Strong Python and SQL skills. Hands-on Databricks experience (DLT, Delta Lake; Unity Catalog desirable). Solid understanding of data modelling, data warehousing, and distributed systems. Experience with cloud data platforms (Azure preferred ...

post include; Develop, construct, test and maintain data architectures within large scale data processing systems. Develop and manage data pipelines using Azure Data Factory, Delta Lake and Spark. Utilise Azure Cloud architecture knowledge to design and implement scalable data solutions. Utilise Spark, SQL, Python, R, and other data ...

responding to business and project needs rather than operating as a narrow silo. You'll work hands-on with Azure Databricks, Azure Data Factory, Delta Lake, and Power BI to create scalable data models, automated pipelines, and self-service analytics capabilities. This is a fantastic opportunity to join ...

intuition-clean layouts, drill paths, and KPI logic. Nice to Have Python for automation or ad-hoc prep; PySpark familiarity. Understanding of Lakehouse patterns, Delta Lake, metadata-driven pipelines. Unity Catalog/Purview experience for lineage and governance. RLS/OLS implementation experience. ...

consumption layers Technology Leadership Evaluate and recommend modern data technologies and tooling Stay current with industry trends such as open table formats (Apache Iceberg, Delta Lake), data observability, and cloud-native services Lead proofs-of-concept and technical assessments for new technologies Ensure the data architecture remains modern ...

intuition-clean layouts, drill paths, and KPI logic. Nice to Have Python for automation or ad-hoc prep; PySpark familiarity. Understanding of Lakehouse patterns, Delta Lake, metadata-driven pipelines. Unity Catalog/Purview experience for lineage and governance. RLS/OLS implementation experience. ...

deliver the organisation’s data vision and roadmap with senior technology leaders. Act as the technical authority for data architecture across Azure, Microsoft Fabric, Delta Lake, and Power BI. Lead the design, build, and maintenance of scalable data products, pipelines, and Lakehouse solutions. Oversee data migration from legacy … building capability and technical standards. Essential Skills & Experience Strong experience building data products and pipelines using Azure data technologies such as Microsoft Fabric or Delta Lake. Skilled in Python, T-SQL, and modern development practices (including test-driven development). Extensive experience designing data architecture, data models, integrations ...

trading workflows. Exposure to cloud platforms (Azure, AWS, or GCP). Experience hiring, mentoring, or leading small engineering teams. Familiarity with Apache Spark, Kafka, Delta Lake, or similar data technologies. Apply Now to avoid disappointment! To find out more about Huxley, please visit ...

business‐critical workloads and is developed for both real‐time and batch processing. The platform makes extensive use of Azure Databricks , Apache Spark, and Delta Lake to support high‐throughput data pipelines. Kafka and Azure Event Hubs are used to implement reliable, event‐driven architectures, with schema management … stream processing forming core parts of the system. Datasets are stored and served primarily using ADLS Gen2 and Delta tables , with additional technologies such as Azure Data Explorer, Cosmos DB, or Synapse used where they are the right fit. Engineers work closely with SQL , Python, and Scala to model ...

site in Bristol 3 days per week. What you’ll be doing Build, enhance and maintain data pipelines using Azure Databricks , Data Factory , and Delta Lake Develop and optimise Lakehouse components and cloud-based data flows Create robust data models to support analytics, MI and downstream reporting Assist ...

Contribute to the overall data strategy and roadmap, ensuring alignment with business objectives. Tech Stack & Skills Required: Strong hands-on experience with Databricks (Spark, Delta Lake, notebooks). Deep knowledge of Azure data services such as: Strong background in Python and/or Scala. Solid understanding ...

Factory pipelines, Lakehouse) or Azure Synapse Proven experience extracting and consolidating master data from multiple ERPs PySpark and/or T-SQL Understanding of Delta Lake, medallion architecture and lakehouse patterns Understanding of data quality, data governance, and best practices Ability to design clear, reliable, and scalable data ...

Role and Responsibility The role focuses on designing, building and maintaining scalable data pipelines and products on a modern Databricks Lakehouse environment, using Delta Lake, Spark, Python (PySpark) and SQL to deliver high-quality, well-governed data across the business. You'll manage and optimise the cloud-based ...

freshness. Any AI/ML knowledge specifically around LLM's/RAG would also be useful. Data Engineer Key Skills: Python/Pyspark Databricks Delta Lake Azure LLM's/RAG Location: Remote Rate: Circa £450/Day (Outside) Contract Length: 6-12 Months Initial If this Data ...

freshness. Any AI/ML knowledge specifically around LLM's/RAG would also be useful. Data Engineer Key Skills: Python/Pyspark Databricks Delta Lake Azure LLM's/RAG Location: Remote Rate: Circa £450/Day (Outside) Contract Length: 6-12 Months Initial If this Data ...