151 to 175 of 215 Delta Lake Jobs

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

performance, cost and standards-compliance metrics, using them to guide roadmap priorities and architecture governance decisions. PERSONAL PROFILE Deep expertise in Databricks (Unity Catalog, Delta Lake, Spark, Databricks SQL) or equivalent Lakehouse platforms, including administration, cost management, and performance optimisation. Strong understanding of cloud-native architecture principles, cost ...

creation of data domains that align with the banks underlying capabilities Data Engineering & Platforms Design data product schemas across Snowflake and Databricks, including Delta Lake structures and access patterns Define and enforce architectural guardrails for data product design, including performance, scalability, security, and cost efficiency Set design standards ...

control the creation of data domains that align with the banks underlying capabilitiesData Engineering & PlatformsDesign data product schemas across Snowflake and Databricks, including Delta Lake structures and access patternsDefine and enforce architectural guardrails for data product design, including performance, scalability, security, and cost efficiencySet design standards and review ...

Disclosure and Barring Service)Mandatory SkillsAzure Databricks Platform EngineeringHands-on implementation and troubleshooting of Azure DatabricksDatabricks Serverless configuration and workload optimisationDatabricks SQL and Delta Lake developmentCluster management, compute policies, autoscaling, and workload monitoringData EngineeringStrong Python, PySpark, and SQL development skillsDesign and build ingestion and transformation pipelinesFull and incremental ...

grade data pipelines, and strong experience with Python, Scala, or similar programming languages. Experience working with modern cloud data platforms such as Databricks (preferred), Delta Lake, Snowflake, or BigQuery. Proven experience driving complex cross-functional initiatives from concept through production. Experience leading technical architecture discussions and engineering design ...

analytical modelling and reporting. Familiarity with cloud platforms (Azure or AWS), particularly in hybrid data environments. Experience working with data Lakehouse technologies (e.g. Delta Lake, Databricks, or similar platforms). Be able to travel to programme and customer meetings across the UK as required. Security cleared or ability ...

Excel skills for analytical modelling and reporting.Familiarity with cloud platforms (Azure or AWS), particularly in hybrid data environments.Experience working with data Lakehouse technologies (e.g. Delta Lake, Databricks, or similar platforms).Be able to travel to programme and customer meetings across the UK as required.Security cleared or ability ...

solid understanding of agile delivery practices including CI/CD, Application Resiliency, and Security. Preferred qualifications, capabilities, and skills Production experience with Databricks (Delta Lake, Unity Catalog, MLflow, Databricks Jobs) and workflow orchestration with Apache Airflow. Strong working knowledge of both supervised and unsupervised techniques, including tree-based ...

working through ambiguity. Desirable MLOps excellence: reproducible pipelines, model versioning, CI/CD, observability, and automated evaluation. Data engineering: proficiency with Databricks, Apache Spark, Delta Lake, MLflow, and SQL; experience integrating datasets and maintaining data quality. Model training and optimisation: experience with pre‐training, fine‐tuning, distributed training ...

Experience owning delivery roadmaps, OKRs, and budgets for an architecture or engineering function. Strong technical foundation across the data platform stack: Databricks lakehouse architecture (Delta Lake, medallion patterns), Azure Synapse (star schemas, dedicated pools), and data pipeline orchestration (ADF or Databricks Workflows). Experience with data quality frameworks ...

decisions, and code where it matters most. What you’ll work with Compute & orchestration: Spark, Flink, Beam, Airflow, Dagster, Kafka, PubSub Storage & analytics: data lake and warehouse architectures, Parquet, Iceberg, Delta Lake, BigQuery, Snowflake Metadata & data quality: lineage systems, metadata management, Great Expectations, reproducibility systems Cost & performance ...

Azure Databricks Data Engineer to design, develop, deploy, and optimize scalable data solutions on Microsoft Azure. You will build production-grade data pipelines with Delta Lake, Unity Catalog, and Databricks tools. The role requires hands-on expertise in PySpark, Python, SQL, and Spark architecture, plus experience with GitLab … based CI/CD, Databricks Asset Bundles, and Azure Data Lake Storage. #J-18808-Ljbffr ...

matters? The future of cybersecurity starts with you.About the Team:The data engineering team is on a mission to create a hyper scale data lake, which helps find bad actors and stop breaches. The team builds and operates systems to centralize all of the data the falcon platform collects … from you This team would be responsible for driving an application-based, self-service Data Replay that would leverage data within our security data lake and ensure we provide complete Business Continuity and Resiliency for downstream services or peers that ingest and process data within the CrowdStike Cloud Platform.What ...

will build robust ETL/ELT pipelines with Azure Data Factory, Synapse and Databricks, design real-time and batch ingestion, and modern data platforms (Delta Lake, lakehouse, ADLS Gen2). Hybrid-working is offered. #J-18808-Ljbffr ...

enable analytics, reporting, and AI initiatives across Pricing, Underwriting and Claims. You will design and build scalable data pipelines on Azure Databricks and Delta Lake, with governance, security, and real-time processing at scale. The role sits in the Enterprise Data Team and requires strong SQL and PySpark ...

highly beneficial Ability to quickly adapt to new tools and independently deliver scalable solutionsFamiliarity with data pipelines involving Kafka, Debezium, S3, Lambda, and Delta Lake is a plusApply now to find out more!Location:Newcastle upon Tyne Tyne and Wear Newcastle upon Tyne NE12 8BX Salary:50000 - 70000Sector ...

medallion architecture) Leading the design of cloud data platforms (AWS) - including services like Redshift, S3, Glue, Athena Evaluating and introducing new technologies (eg Iceberg, Delta Lake, observability tools) Driving data governance, standards, and best practice Acting as a trusted advisor to both technical and business stakeholders What ...

observability Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns Production-level PySpark and SQL development skills Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM Experience with Databricks Workflows, orchestration and CI/CD Understanding of monitoring, alerting, replay ...

data platform and services using the below technologies: Python, pandas and SQL in Jupyter notebooks for analysis dbt and Databricks data platform with Delta Lake Lightdash as our BI tool and semantic layer Streamlit for interactive data applications Airflow for orchestration Kubernetes for application deployment CI/ ...

data platform and services using the below technologies: Python, pandas and SQL in Jupyter notebooks for analysis dbt and Databricks data platform with Delta Lake Lightdash as our BI tool and semantic layer Streamlit for interactive data applications Airflow for orchestration Kubernetes for application deployment CI/ ...

Cycle, and a solid understanding of agile delivery practices including CI/CD, Application Resiliency, and Security.Preferred qualifications, capabilities, and skillsProduction experience with Databricks (Delta Lake, Unity Catalog, MLflow, Databricks Jobs) and workflow orchestration with Apache Airflow.Experience with evaluation frameworks and approaches (golden datasets, LLM-as-judge, human ...

covering the full lifecycle, discovery, design and build, through to production release and support Deep hands‐on experience of the core platform: Unity Catalog, Delta Lake, workflows and jobs, declarative pipelines, SQL warehouses, cluster and compute configuration Migrated data at scale off on‐prem or legacy platforms ...

Python, Data Modelling, Data Warehousing.Strong background in incident resolution, requests, changes and problem‐solving within SLAs.Hands‐on experience with Spark, Data Architecture, SQL and Delta Lake.Cloud development experience (Azure/Fabric).Working knowledge of Medallion Architecture.Demonstrated capability in implementing and supporting pipelines in demanding environments.Strong communication skills and confidence ...

practices, and escalating, managing and reporting control issues with transparency.Skills & ExperienceEssentialProven in-depth commercial experience in the following key areas:Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning.Apache Spark & Scala – High-performance distributed data processing and optimisation.Cloud Data Engineering (AWS) – Cloud-native data platforms and distributed … Applications DevelopmentDemonstrated knowledge of client core business functionsDemonstrated leadership, project management, and development skillsRelationship and consensus building skillsPreferredKafka/Event-Driven ArchitectureApache Iceberg/Delta LakePythonAI/ML & LLM integrationFinancial Services/Capital MarketsEducation:Bachelor’s degree/University degree or equivalent experienceMaster’s degree preferredWhat we’ll provide ...