76 to 100 of 322 Permanent PySpark Jobs

Description What you'll do Data Engineering Delivery: Design, build andoptimisescalable data pipelines, ETL/ELT workflows and data transformation solutions using Databricks,PySpark, SQL anddbtCloud, delivering trusted, high-quality data products that support reporting, analytics, AI and data science. Data Architecture & Modelling: Develop and maintain robustlakehousearchitectures and reusable ...

depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. Deep lakehouse and data platform ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

API Gateway, and CloudWatch. Design, develop, and maintain data pipelines and ETL workflows using technologies such as Glue, Spark/PySpark, Kafka, or similar platforms. Implement and maintain Infrastructure as Code (IaC) using Terraform or equivalent tools. Drive observability, monitoring, logging, and operational readiness across software platforms and services. ...

Data Engineering roles, including 3 years in a technical lead role managing end‐to‐end solutions. Hands‐on experience with Azure Databricks using PySpark, with the ability to undertake cluster capacity planning and workload optimisation. Holds Security Check (SC) clearance OR Eligible for Security Check (SC) clearance level ...

platform or data warehouse migration (cutover, reconciliation, parallel running, decommissioning) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

Generation), Serverless, DWH models, Data Mesh, Data Fabric, Medallion and Data Products. Hands on mindset : Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON. Agile Ways of Working Run and contribute to agile ceremonies, helping to shape ...

Data Engineer Key skills: * Microsoft Fabric * Azure Cloud Technologies * Python * SQL Server/SQL * Data Warehousing & Data Modelling * Data Architecture * Data Pipelines & Integration Solutions * PySpark * ETL/ELT Development * Data Governance & Security * Source Control (Git) * Technical Leadership & Team Management You will be a Lead Data Engineer Developer with good ...

data warehousing, dimensional modelling, Kimball principles and data architecture Advanced SQL/T-SQL skills, including query development, optimisation and troubleshooting Strong Python, PySpark or similar programming skills for data manipulation and automation Experience integrating data from APIs, databases, files and other sources Experience with data quality, pipeline monitoring ...

strong problem solving, communication, and stakeholder management skills. Requirements 4-6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven experience designing and implementing metadata-driven data pipelines. ...

problem solving, communication, and stakeholder management skills.RequirementsCandidates should have:4-6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions.Proven experience designing and implementing metadata-driven data pipelines.Strong understanding ...

workflows Defining and enforcing standardised engineering patterns across a shared codebase Operational reliability, including monitoring, structured logging, alerting, and systematic failure recovery Desirable PySpark Experience supporting Airflow in a self‐managed environment Experience in a business building a data platform from the ground up Working Arrangements This ...

record of development in Databricks* Proven track record of working with IaC* Proven track record of working with git* Strong working knowledge of Python (Pyspark) and SQL* Strong working knowledge of data management inside a datalake* Proven ability to explain technical issues and solutions to peers and customers* Proven ...

record of development in Databricks* Proven track record of working with IaC* Proven track record of working with git* Strong working knowledge of Python (Pyspark) and SQL* Strong working knowledge of data management inside a datalake* Proven ability to explain technical issues and solutions to peers and customers* Proven ...

record of development in Databricks* Proven track record of working with IaC* Proven track record of working with git* Strong working knowledge of Python (Pyspark) and SQL* Strong working knowledge of data management inside a datalake* Proven ability to explain technical issues and solutions to peers and customers* Proven ...

record of development in Databricks Proven track record of working with IaC Proven track record of working with git Strong working knowledge of Python (Pyspark) and SQL Strong working knowledge of data management inside a datalake Proven ability to explain technical issues and solutions to peers and customers Proven ...

data ingestion processes from clinical, operational, financial and external systems, including relational databases, REST APIs and file-based sources. Build and optimise SQL and PySpark transformations across Bronze, Silver and Gold data layers. Work with technologies including Databricks, Delta Lake, Lakeflow, dbt and Unity Catalog. Implement automated data quality … similar data engineering role. Strong SQL skills, including performance tuning, CTEs, window functions and data modelling. Good Python programming skills, particularly PySpark, for data transformation, API integrations and automation. Hands-on experience with Databricks, including Delta Lake, Spark and Unity Catalog, or a comparable cloud data platform such ...

responsibilities. Ensure solutions are secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines … duties as required. Skills and Experience Strong understanding of data engineering principles, data architecture, and cloud-based data platforms. Proficient in Python (PySpark) and SQL for building and optimising data transformations at scale. Strong hands-on Databricks experience. Experience with Azure data storage technologies (Azure SQL, Data Lakes, Blob ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

Looking For Proven expertise with Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms. Extensive experience implementing CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions ...

looking for: Previous experience as a Technical Lead (hands-on) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environments Strong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution‐building skills Solid grounding in distributed data systems architecture: scalable pipelines, streaming architectures, lakehouse ...

security, and governance to support AI agent-driven data products. The ideal candidate has deep SQL skills (SQL Server/Azure SQL), strong Python (PySpark), and experience with Databricks, Synapse or Fabric, plus Power BI semantic modelling. A background in data security and Agile delivery is preferred. #J ...

agents that underpin our data-driven decisions. The role requires deep SQL/T-SQL on SQL Server/Azure SQL, strong Python (PySpark) capabilities, and experience with Databricks, Synapse or Microsoft Fabric, plus Power BI semantic modelling. #J-18808-Ljbffr ...