26 to 50 of 142 PySpark Jobs in London

relevant discipline or equivalent practical experience. Desirable Experience Strong hands-on SQL and Python skills, with experience in distributed data processing using PySpark and/or Databricks. Experience with AWS services, APIs, event-driven integration and infrastructure as code. Exposure to AI/ML data preparation, data observability, metadata ...

degree in a relevant discipline or equivalent practical experience. Strong hands-on SQL and Python skills, with experience in distributed data processing using PySpark and/or Databricks. Experience with AWS services, APIs, event-driven integration and infrastructure as code. Exposure to AI/ML data preparation, data observability ...

Looking For Proven hands-on experience with the Databricks platform, including orchestration and enterprise-scale Azure data engineering solutions. Strong technical skills in Python, PySpark, and SQL, with experience working with distributed data processing frameworks. Expertise delivering end-to-end data pipelines across ingestion, transformation, and serving layers within ...

/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft Purview. ...

Data Analytics/Warehousing & Business Intelligence Experience 5-10 years Location United Kingdom - London Skills Python & SQL: Advanced Python (PySpark, Advanced SQL (CTEs, Databricks We are looking for an experienced Principal Data Engineer to take technical ownership of our customer decisioning pipeline architecture, end-to-end orchestration, and production ...

only on ETL. However, we will consider candidates who bring Cloud + CI/CD experience alongside technologies such as: Proficient SQL; familiarity with PySpark is beneficial dbt Microsoft Fabric SQLMesh 5. Core Engineering Skills Strong Python, with good documentation and unit testing practices Comfortable in an agile delivery ...

architecture Experience with cloud data platforms (AWS preferred - Redshift, Athena, S3, Glue. Azure accepted) Understanding of workflow orchestration and automation tools Python experience (including PySpark) is beneficial but not essential Exposure to BI platforms such as Power BI, Tableau or QuickSight, with an understanding of how backend data models ...

successfully managing and leading multi-disciplinary engineering teams. Solid understanding of the software and data delivery lifecycle. Technical Skills Strong experience with Databricks (PySpark/Spark SQL, Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows). Working knowledge of Microsoft Dynamics 365 data structures, Dataverse, and common integration patterns ...

database technologies including data lakes such as Azure SQL or SQL Server, PostgreSQL, Snowflake, MySQL, Cosmos DB or Mongo DB with Python and Pyspark Building data warehouse or data lakes with SCVs or in conjunction with CDP platforms Experience of ELT/ETL tools such as Databricks, Azure Data ...

experience in data engineering, including 3+ years in a tech lead capacity Strong hands-on expertise in Databricks for enterprise-scale workloads Proficiency in PySpark, Spark Structured Streaming, and Delta Lake Solid programming skills in Python and demonstrated SQL performance optimization experience Deep understanding of Azure Data Platform services ...

patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data ...

Databricks Apps. Databricks certification at Professional level, or Associate level with the drive to progress (see Certifications). Technical skills Python and SQL, including PySpark and Spark SQL; scripting in Bash. Infrastructure as code with Terraform (Databricks and AzureRM providers), plus Declarative Automation Bundles, the Databricks ...

depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. Deep lakehouse and data platform ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

payment, and transaction analysis. Strong Python skills for statistical modelling, cohort analysis, weighting, matching, and reproducible analytics. Hands-on experience with Databricks, Spark/PySpark, Delta tables, and automated data workflows. Proficiency with Git, Azure DevOps, and controlled production release processes. Strong data quality, reconciliation, and audit-trail practices. ...

strong problem solving, communication, and stakeholder management skills. Requirements 4-6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven experience designing and implementing metadata-driven data pipelines. ...

problem solving, communication, and stakeholder management skills.RequirementsCandidates should have:4-6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions.Proven experience designing and implementing metadata-driven data pipelines.Strong understanding ...

workflows Defining and enforcing standardised engineering patterns across a shared codebase Operational reliability, including monitoring, structured logging, alerting, and systematic failure recovery Desirable PySpark Experience supporting Airflow in a self‐managed environment Experience in a business building a data platform from the ground up Working Arrangements This ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

Looking For Proven expertise with Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms. Extensive experience implementing CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions ...

years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environments Strong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution‐building skills Solid grounding in distributed data systems architecture: scalable pipelines, streaming architectures, lakehouse ...

calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines. Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest) , PySpark , and advanced SQL to catch discrepancies, nulls, and duplicate records. CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure ...

best practices and driving a culture of continuous improvement. Requirements 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven track record of designing and implementing metadata‐driven data pipelines. ...

Databricks experience to build, develop and manage STACK's enterprise data platform. This is a production-focused engineering role requiring practical expertise in Databricks, PySpark, Python, SQL and Delta Lake. The role will design reliable pipelines integrating Finance, Delivery, Operations, HR and PMO systems, establish governed and high-quality … help improve the reliability and performance of the platform. Key Responsibilities: Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets. Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting ...