101 to 125 of 344 PySpark Jobs

Looking For Proven expertise with Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms. Extensive experience implementing CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions ...

looking for: Previous experience as a Technical Lead (hands-on) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environments Strong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution‐building skills Solid grounding in distributed data systems architecture: scalable pipelines, streaming architectures, lakehouse ...

security, and governance to support AI agent-driven data products. The ideal candidate has deep SQL skills (SQL Server/Azure SQL), strong Python (PySpark), and experience with Databricks, Synapse or Fabric, plus Power BI semantic modelling. A background in data security and Agile delivery is preferred. #J ...

agents that underpin our data-driven decisions. The role requires deep SQL/T-SQL on SQL Server/Azure SQL, strong Python (PySpark) capabilities, and experience with Databricks, Synapse or Microsoft Fabric, plus Power BI semantic modelling. #J-18808-Ljbffr ...

Data Lakehouse solutions AWS services including S3, Glue, EMR, Athena, Redshift, Lake Formation, EventBridge & Step Functions Advanced Python and SQL skills Strong experience with PySpark and large-scale data processing Hands-on experience with dbt or modern data transformation frameworks Data modelling experience (Kimball, Data Vault, Inmon) Strong understanding ...

calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines. Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest) , PySpark , and advanced SQL to catch discrepancies, nulls, and duplicate records. CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure ...

week in office ( remote might be considered) Length: 6+ months, might be prolonged Interview: GI TI PI - CI Must have: Snowflake + SnowPark or PySpark + SQL + experience with any ML capabilities JD: Data Engineering work to support Design of USO Patient Journey Optimization and Pre-Call Intelligence ...

best practices and driving a culture of continuous improvement. Requirements 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven track record of designing and implementing metadata‐driven data pipelines. ...

party data providers, delivering reliable and auditable data at scale. Key Responsibilities Build and maintain ingestion and transformation pipelines on Microsoft Fabric (Data Pipelines, PySpark notebooks, Dataflows Gen2) across the Landing, Curate and Enrich layers of the medallion architecture. Onboard new source systems by setting up the team … Snowflake and SQL Server, change data capture, file and API loads). Do this through configuration rather than one-off code. Write and tune PySpark and T-SQL for Lakehouse and Warehouse workloads, including fact and dimension models in the enterprise data warehouse. Implement data quality rules and validation ...

Databricks experience to build, develop and manage STACK's enterprise data platform. This is a production-focused engineering role requiring practical expertise in Databricks, PySpark, Python, SQL and Delta Lake. The role will design reliable pipelines integrating Finance, Delivery, Operations, HR and PMO systems, establish governed and high-quality … help improve the reliability and performance of the platform. Key Responsibilities: Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets. Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting ...

growth across regulated industries.ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules, logging and alerting to data pipelinesBuild enterprise semantic models (star ...

across Financial Services, Fintech, Public Sector and Utilities. You will work on modern data engineering, reporting, and cloud data solutions with Python, Spark, SQL, PySpark and Power BI. The role emphasizes client-facing advisory, Agile delivery, and experience with Snowflake/Databricks in a dynamic data practice. #J ...

London area will be required, with a flexible hybrid approach that supports remote work. Candidates should have hands-on experience with Python, Spark, SQL, PySpark, and PowerBI, and be comfortable working across AWS/Azure/GCP and Snowflake/Data Bricks. SC clearance eligibility is required. #J ...

data analysis Strong Databricks experience including notebooks, modules, jobs, workflows, Delta tables, Unity Catalog data, model registry, and production scoring Hands‐on Spark or PySpark capability for analytical pipelines and large-scale data processing Git and Azure DevOps including branching, pull requests, code review, testing, and controlled deployments Strong ...

Essential Skills: Data pipeline development and optimisation Data modelling and architecture (in Azure Databricks) Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark) Data science and machine learning Data governance and quality assurance Problem-solving and analytical thinkingStakeholder engagement and technical communication, including at design forums where ...

Essential Skills:*** Data pipeline development and optimisation* Data modelling and architecture (in Azure Databricks)* Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark)* Data science and machine learning* Data governance and quality assurance* Problem-solving and analytical thinking* Stakeholder engagement and technical communication, including at design forums ...

payment, loyalty and transaction analysis. Strong Python for statistical analysis, matching, weighting, cohort analysis and reproducible data products. Hands-on Databricks/Spark or PySpark experience for notebooks, Delta tables, workflows and scheduled analytical pipelines. Working proficiency with Git/Azure DevOps, pull requests and controlled production changes. Strong ...

. Deep expertise in modern data platforms optimized for ML workloads (Databricks, Auto ML, Snowflake, BigQuery). Strong proficiency in Python, SQL, Spark/PySpark for large-scale data processing. Experience with data orchestration tools (Airflow, Prefect, dbt) and CI/CD for data pipelines. Understanding of data governance ...

ingest, cleanse and model data for BI reporting, ML and AI, ensuring quality and governance. The role emphasizes strong Azure skills (ADF, ADLS, Databricks), PySpark, Python and SQL, with a DevOps and CI/CD mindset. Hybrid work with three days in the office weekly is offered. #J ...

investigate complex problems and take ownership of technical solutions. Strong communication skills and a collaborative approach. Desirable Experience Data engineering technologies such as Databricks, PySpark or Spark. Cloud platforms, particularly AWS. Python frameworks such as FastAPI and PyTorch. Experience developing APIs and data pipelines. Infrastructure as Code, CI/ ...

services, and scripting and automation using common libraries and frameworks relevant to the role Experience with data engineering and analytics platforms such as PySpark, Databricks, Airflow, or similar orchestration tools Experience with relational databases and SQL; familiarity with NoSQL databases and data modeling concepts Knowledge across the data lifecycle ...

agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL.Solid understanding of data engineering, data modelling, security and governance in enterprise settings. Experience with software engineering best practices including ...

experience: Proven experience building production-grade data pipelines and ETL processes. Excellent Python skills with experience developing scalable data solutions. Experience working with PySpark or distributed data processing frameworks. Experience using GitHub and modern software engineering practices. Understanding of cloud-based data platforms, ideally Azure. Experience working with sensitive ...