Principal Data Platform Architect
We are seeking a Principal Data Platform Architect with proven experience leading enterprise data platform migrations from Palantir Foundry to Databricks. This role will drive the strategic design, migration, and optimization of the enterprise data ecosystem, transitioning data pipelines, ontologies, and transformation workflows into a scalable Databricks Lakehouse architecture.
The successful candidate will provide architectural leadership and technical direction, ensuring data integrity, governance, performance, and long-term sustainability. You will collaborate closely with data engineering, DevOps, and business stakeholders to modernize the platform, establish best practices, and deliver a secure, high-performance migration.
Responsibilities:
- Define and execute the end-to-end migration roadmap from Palantir Foundry to Databricks, including target architecture, data models, security, and governance frameworks.
- Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL.
- Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration.
- Optimize Spark workloads, cluster configurations, and storage strategies for high performance and cost efficiency.
- Collaborate with data engineers, platform teams, and business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability.
Mandatory Skills:
- Advanced Databricks experience: architecture, Delta Lake, job orchestration, cluster management, performance tuning.
- Proven migration experience (Palantir → Databricks) necessary, including pipeline conversion, data model redesign, and production cutover.
- Expert-level PySpark & Python skills for distributed data processing workflows.
- Advanced SQL & data modelling expertise (dimensional modelling, lakehouse patterns, query optimization).
- Cloud platform experience (Azure preferred) including storage, security, and networking.
Nice to Have:
- Hands-on expertise in Palantir Foundry (pipelines, ontologies, lineage, transformations, governance).
- Understanding of the investment data domain.
- Familiarity with Dynatrace or Datadog for observability and monitoring.
- Certifications: Databricks, Azure/AWS, enterprise data architecture.
Join us to lead a strategic migration from Palantir Foundry to Databricks, modernizing enterprise data platforms for scalability, governance, and performance. Apply today to bring your architectural expertise to a high-impact transformation.
No sponsorship