1 to 25 of 59 Permanent Delta Lake Jobs

governed, trusted BI products from model to report layer. Architect the platform — Lead the shift to a cloud-native lakehouse on Azure Databricks with Delta Lake and Unity Catalog. Establish Python-first standards and reusable framework patterns for the team. Rationalise KPIs — Drive "one version of the truth … fluency across the modern Azure data stack — and strong bridging skills for the legacy environment you're migrating from. Python, PySpark, Azure Databricks (Delta Lake, Unity Catalog, pipelines, notebooks, jobs), Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Power BI (complex model design, advanced ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

governance controls and operational guardrails Technical Leadership & Standards Serve as the senior technical authority for the Databricks platform architecture Establish engineering best practices covering Delta Lake design, workload orchestration, compute optimization, and cost management Provide technical assurance across platform implementation activities delivered by internal teams and external partners … communication, stakeholder engagement, and consulting capabilities Experience translating business requirements into strategic platform architecture solutions Technical Skills Deep expertise in Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and cluster management Strong experience designing cloud-native data platforms on AWS Hands‐on knowledge of AWS services including ...

Data Engineer will design, build, support and operate data pipelines and core data platform technologies within the organisation's Azure-based analytics and lake house environment. The role is hands-on and implementation-focused, working within established architectural patterns to deliver reliable, secure and well-governed data for analytics … Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including performance monitoring and configuration Build Gold-layer datasets optimised for Power ...

architecture skills, customer-facing experience, and the ability to connect advanced technology solutions with business objectives. Design scalable Databricks solution architectures based on Lakehouse, Delta Lake, and Unity Catalog patterns aligned with customer needs. Lead technical discovery sessions to understand business objectives, data challenges, and existing technology environments. … customer-facing technical positions such as solutions architecture, technical consulting, or sales engineering. Proven experience designing and delivering enterprise data architectures using Lakehouse, Delta Lake, and Unity Catalog patterns. Experience working with cloud platforms including AWS, Azure, and/or GCP. Databricks Professional-level certification, such as Data ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Azure and Databricks . Key Responsibilities Design and implement scalable, secure cloud‐based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real‐time analytics and machine learning use cases … internal stakeholders and third‐party vendors Key Skills & Experience Proven background designing and delivering enterprise‐scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

managing and reporting control issues with transparency.**Skills & Experience****Essential**Proven in-depth commercial experience in the following key areas:* Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning.* Apache Spark & Scala – High-performance distributed data processing and optimisation.* Cloud Data Engineering (AWS) – Cloud-native data platforms … core business functions* Demonstrated leadership, project management, and development skills* Relationship and consensus building skills**Preferred*** Kafka/Event-Driven Architecture* Apache Iceberg/Delta Lake* Python* AI/ML & LLM integration* Financial Services/Capital Markets**Education:*** Bachelor’s degree/University degree or equivalent experience* Master ...

risks, exceptions, and technical dependencies raised through governance processes Enterprise & Solution Architecture Provide architecture assurance for Databricks Lakehouse implementations, including data ingestion, medallion architecture, Delta Lake, Unity Catalog, and compute strategies Validate integration patterns across Databricks, cloud platforms, enterprise applications, and data ecosystems Define and maintain reference architectures … delivery requirements Exceptional stakeholder engagement and influencing skills across technical and business audiences Technical Skills Strong hands‐on knowledge of Databricks including Lakehouse Architecture, Delta Lake, Unity Catalog, and platform governance Experience with cloud platforms supporting Databricks environments, including AWS and/or Azure Solid understanding of enterprise ...

deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products. The role emphasizes batch/streaming pipelines, Spark/PySpark, Delta Lake, governance and security, with CI/CD and collaboration across environments. #J-18808-Ljbffr ...

Skills Strong experience designing and delivering modern data platforms using Microsoft Fabric, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL and Data Lake technologies . Proven experience implementing Lakehouse, Data Warehouse and Medallion Architecture patterns, including data modelling, governance and performance optimization . Expertise in building scalable … Python, Spark and PySpark for data engineering, analytics and automation workloads. Experience with Azure Databricks and/or Microsoft Fabric engineering workloads, including notebooks, Delta Lake, data transformation, optimization and operationalization of data solutions. An understanding of data governance, security, lineage and data quality, using tools and frameworks ...

migration (6 months), then platform evolution, DevOps maturity, and capability expansion Core Technical Requirements Azure Data Platform Architecture Microsoft Azure data stack expertise: ADF, Delta Lake, ADLS Gen2, Synapse, Databricks (with flexibility to integrate non-Microsoft solutions) Experience with PostgreSQL and SQL Server, including data replication/integration … Python and SQL for ELT/ETL pipelines Batch, streaming, and micro-batching architectures CDC patterns and incremental load strategies at scale Experience with Delta Change Data Feed or equivalent real-time sync mechanisms Experience building ingestion pipelines from diverse external APIs (CRM, analytics, finance, observability tools) into ...

environment strategy. Design and govern data integration patterns for sharing data with external clients and third parties, including secure, scalable approaches using Databricks Delta Sharing, APIs, and file-based exchange where appropriate. Lead the design and adoption of Master Data Management (MDM), including entity definitions, golden-record strategy, match … merge rules, and stewardship processes. Establish standards and practices for optimizing Databricks performance and cost — cluster and compute strategy, Delta table management, caching, and workload right-sizing. Define data modelling standards (conceptual, logical, physical) across analytical and operational domains. Collaborate with data governance to embed data quality, lineage, cataloguing ...

files, and event streams Supporting master data management, data quality rules, and golden record definition Working hands-on with ADLS, Azure SQL, Event Hubs, Delta Lake, and OneLake Designing integration patterns and REST APIs to support near real-time data flows Contributing to infrastructure-as-code and environment ...

strategy to hands‐on delivery. Experience in a startup or small‐team environment , comfortable wearing multiple hats. Nice‐to‐Have Experience with Power BI , Delta Lake , Synapse , or Databricks . Knowledge of Python or TypeScript for data or automation scripts. Familiarity with CI/CD pipelines , IaC (Terraform ...

skills – you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environment Experience in the financial markets, especially in delta one, store of value, and/or FICC options trading Experience with Linux‐based, concurrent, high‐throughput, low‐latency software systems Experience with pipeline … orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information about DRW’s processing activities ...

expectations; ability to guide peers on safe and effective usage within team practices Preferred Qualifications, Capabilities, and Skills: Experience with Databricks lakehouse, Databricks Genie, Delta Lake, and medallion architecture ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

model lifecycle: data prep, experimentation, deployment, and monitoring.* Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS.* Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing.* Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering.* Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering.* Experience with Databricks, Spark, and Delta Lake.* Strong SQL skills with experience working on large datasets.* Experience with AWS, including S3 and Lambda.* Familiarity with Redshift, Snowflake, or other cloud ...

delivering large scale Data science or AI Projects Strong knowledge of LLMs (GPT, Llama, Claude, Mistral) and their reasoning capabilities Deep Databricks experience (DLT, Delta Lake, Unity Catalog governance) Solid grasp of streaming technologies (Spark Structured Streaming, Autoloader) Python, SQL, or Scala proficiency Experience with ETL/ ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. ...

cloud databases Proven track record with complex data migration projects (terabyte+ datasets, multiple legacy source systems, structures and unstructured data) Proficiency with Parquet/Delta Lake or other modern data storage formats Experience with streaming architectures using Kafka, Event Hubs, or Kinesis for real-time data processing Knowledge ...