101 to 125 of 322 Permanent PySpark Jobs

Data Lakehouse solutions AWS services including S3, Glue, EMR, Athena, Redshift, Lake Formation, EventBridge & Step Functions Advanced Python and SQL skills Strong experience with PySpark and large-scale data processing Hands-on experience with dbt or modern data transformation frameworks Data modelling experience (Kimball, Data Vault, Inmon) Strong understanding ...

calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines. Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest) , PySpark , and advanced SQL to catch discrepancies, nulls, and duplicate records. CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure ...

week in office ( remote might be considered) Length: 6+ months, might be prolonged Interview: GI TI PI - CI Must have: Snowflake + SnowPark or PySpark + SQL + experience with any ML capabilities JD: Data Engineering work to support Design of USO Patient Journey Optimization and Pre-Call Intelligence ...

best practices and driving a culture of continuous improvement. Requirements 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven track record of designing and implementing metadata‐driven data pipelines. ...

party data providers, delivering reliable and auditable data at scale. Key Responsibilities Build and maintain ingestion and transformation pipelines on Microsoft Fabric (Data Pipelines, PySpark notebooks, Dataflows Gen2) across the Landing, Curate and Enrich layers of the medallion architecture. Onboard new source systems by setting up the team … Snowflake and SQL Server, change data capture, file and API loads). Do this through configuration rather than one-off code. Write and tune PySpark and T-SQL for Lakehouse and Warehouse workloads, including fact and dimension models in the enterprise data warehouse. Implement data quality rules and validation ...

Databricks experience to build, develop and manage STACK's enterprise data platform. This is a production-focused engineering role requiring practical expertise in Databricks, PySpark, Python, SQL and Delta Lake. The role will design reliable pipelines integrating Finance, Delivery, Operations, HR and PMO systems, establish governed and high-quality … help improve the reliability and performance of the platform. Key Responsibilities: Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets. Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting ...

growth across regulated industries.ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules, logging and alerting to data pipelinesBuild enterprise semantic models (star ...

across Financial Services, Fintech, Public Sector and Utilities. You will work on modern data engineering, reporting, and cloud data solutions with Python, Spark, SQL, PySpark and Power BI. The role emphasizes client-facing advisory, Agile delivery, and experience with Snowflake/Databricks in a dynamic data practice. #J ...

London area will be required, with a flexible hybrid approach that supports remote work. Candidates should have hands-on experience with Python, Spark, SQL, PySpark, and PowerBI, and be comfortable working across AWS/Azure/GCP and Snowflake/Data Bricks. SC clearance eligibility is required. #J ...

data analysis Strong Databricks experience including notebooks, modules, jobs, workflows, Delta tables, Unity Catalog data, model registry, and production scoring Hands‐on Spark or PySpark capability for analytical pipelines and large-scale data processing Git and Azure DevOps including branching, pull requests, code review, testing, and controlled deployments Strong ...

Essential Skills: Data pipeline development and optimisation Data modelling and architecture (in Azure Databricks) Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark) Data science and machine learning Data governance and quality assurance Problem-solving and analytical thinkingStakeholder engagement and technical communication, including at design forums where ...

Essential Skills:*** Data pipeline development and optimisation* Data modelling and architecture (in Azure Databricks)* Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark)* Data science and machine learning* Data governance and quality assurance* Problem-solving and analytical thinking* Stakeholder engagement and technical communication, including at design forums ...

payment, loyalty and transaction analysis. Strong Python for statistical analysis, matching, weighting, cohort analysis and reproducible data products. Hands-on Databricks/Spark or PySpark experience for notebooks, Delta tables, workflows and scheduled analytical pipelines. Working proficiency with Git/Azure DevOps, pull requests and controlled production changes. Strong ...

. Deep expertise in modern data platforms optimized for ML workloads (Databricks, Auto ML, Snowflake, BigQuery). Strong proficiency in Python, SQL, Spark/PySpark for large-scale data processing. Experience with data orchestration tools (Airflow, Prefect, dbt) and CI/CD for data pipelines. Understanding of data governance ...

ingest, cleanse and model data for BI reporting, ML and AI, ensuring quality and governance. The role emphasizes strong Azure skills (ADF, ADLS, Databricks), PySpark, Python and SQL, with a DevOps and CI/CD mindset. Hybrid work with three days in the office weekly is offered. #J ...

investigate complex problems and take ownership of technical solutions. Strong communication skills and a collaborative approach. Desirable Experience Data engineering technologies such as Databricks, PySpark or Spark. Cloud platforms, particularly AWS. Python frameworks such as FastAPI and PyTorch. Experience developing APIs and data pipelines. Infrastructure as Code, CI/ ...

services, and scripting and automation using common libraries and frameworks relevant to the role Experience with data engineering and analytics platforms such as PySpark, Databricks, Airflow, or similar orchestration tools Experience with relational databases and SQL; familiarity with NoSQL databases and data modeling concepts Knowledge across the data lifecycle ...

agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL.Solid understanding of data engineering, data modelling, security and governance in enterprise settings. Experience with software engineering best practices including ...

experience: Proven experience building production-grade data pipelines and ETL processes. Excellent Python skills with experience developing scalable data solutions. Experience working with PySpark or distributed data processing frameworks. Experience using GitHub and modern software engineering practices. Understanding of cloud-based data platforms, ideally Azure. Experience working with sensitive ...

Lake and Lakehouse architecture for efficient data management; experience with ETL processes and optimizing data pipelines for performance Be strong in Python, SQL, and PySpark Be passionate about building data products for stakeholders and customers, ensuring they are stable, scalable, secure, accurate, observable, and performant Enjoy collaborating closely with ...

audiences Essential knowledge, skills & experience (must-have): 4+ years of commercial experience in Data Science/Machine Learning Hands-on experience with: Databricks Notebooks PySpark Workflows Deployment through Asset Bundles Proven experience building, deploying, and maintaining production ML solutions Broad experience across multiple ML domains, including: Forecasting/Time ...

Gen2), Purview, Cosmos DB, OpenAI, Azure ML, and AI Foundry Solid understanding of software engineering concepts, with hands‐on experience using Python, Scala or PySpark, data modelling techniques, database technologies, and SQL Confident communicator, able to explain technical concepts to non‐technical audiences and mentor junior colleagues For Senior ...

Gen2), Purview, Cosmos DB, OpenAI, Azure ML, and AI Foundry Solid understanding of software engineering concepts, with hands‐on experience using Python, Scala or PySpark, data modelling techniques, database technologies, and SQL Confident communicator, able to explain technical concepts to non‐technical audiences and mentor junior colleagues For Senior ...

insights clearly to both technical and non-technical audiences. Some other highly valued skills include: Strong SQL and database experience. Knowledge of Python/PySpark and data analytics. Experience working in a Call Centre or Contact Centre environment. Strong communication and stakeholder management skills. Ability to lead technical teams ...

What will you need to succeed? Strong production‐level Python engineering experience, with the ability to build maintainable, tested, and supportable services. Experience with PySpark or equivalent large‐scale data processing frameworks. Hands‐on experience designing and operating CI/CD pipelines using Azure DevOps or similar tooling. Experience ...