1 to 25 of 247 PySpark Jobs in the UK

transformation and orchestration pipelines, integrating data from cloud, on‐premises and third‐party business systems. Strong hands‐on capability with SQL, Python, Spark and PySpark for data engineering, analytics and automation workloads. Experience with Azure Databricks and/or Microsoft Fabric engineering workloads, including notebooks, Delta Lake, data transformation ...

looking for an experienced Data Engineer with proven expertise in building pipelines in Databricks and a strong grasp of Apache Spark, whether in PySpark or Scala, including Structured Streaming. You should have experience with Kafka (MSK) and real-time data ingestion, as well as a deep understanding of Delta ...

complex data workflows Excellent communication and collaboration skills in agile, cross-functional teams Ability to mentor junior engineers Nice to have: Experience with Databricks, PySpark, and Delta Lake Условия Competitive salary based on location, experience, and skills Comprehensive benefits package for employees and their families Continuous learning, structured development ...

standards and best practices. About you Extensive experience designing and delivering enterprise-scale data engineering solutions Strong hands-on experience with Azure Databricks, including PySpark, Delta Live Tables and Databricks Workflows and orchestration Microsoft Fabric experience, including Lakehouse, Data Factory, Warehouse and Spark Knowledge of Azure Data Platform services ...

understanding on Master Data Services, DQ profiling, cleansing, update, DQ reports, Purview. Strong experience with at least one of the programming languages (Python, Scala, PySpark). Knowledge and experience using the Microsoft technology stack and associated development tools. Experience working with SharePoint, PowerPivot, PowerBI and deep knowledge ...

using Azure Databricks. Build robust ETL/ELT solutions using Azure Data Factory and Microsoft Fabric. Develop high-performance data processing solutions using Python, PySpark, and SQL. Implement Delta Lake and Lakehouse architectures following industry best practices. Work with Azure Data Lake Storage (ADLS Gen2) to manage large-scale ...

with Microsoft Fabric technologies including OneLake, Lakehouse, Warehouse, Data Factory, Spark and Power BI. Strong Engineering Skills - You're proficient in SQL, Python and PySpark, with experience developing scalable and maintainable data solutions. Modern Data Platform Experience - You've successfully built, enhanced or operated cloud-based data platforms, ideally ...

would also be highly advantageous, particularly: Azure Data Factory Azure SQL Azure Data Lake/ADLS Azure DevOps Microsoft Fabric Azure Databricks Experience with PySpark, Airflow, Dagster, Prefect or other orchestration technologies would also be useful. We're particularly interested in people who have worked with modern data platforms ...

Azure Databricks, Databricks, Azure Data Lake, ADLS, Azure SQL, Azure Synapse, Synapse Analytics, Azure PaaS, ETL, ELT, Data Pipelines, Data Integration, SQL, Python, PySpark, Azure DevOps, CI/CD, Data Governance, Data Quality, Cloud Security, Microsoft Fabric, Power BI, Power Platform. ...

/ELT pipelines and cloud data platforms Strong SQL skills for data validation and reconciliation Experience with: PyTest REST API testing tools Python or PySpark GitLab/Azure DevOps pipelines Experience validating large-scale datasets and data transformations Knowledge of data warehousing concepts and dimensional modeling Understanding of Delta ...

architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture Hybrid cloud environments Enterprise integration patterns Modern data formats including Parquet, Avro, and ORC DevOps & Platform ...

complex Power BI reports and dashboards. Proficiency in writing complex DAX formulas and strong understanding of Power Query. Programming Skills: Requires strong Python/PySpark for data manipulation, analysis, and automation, alongside advanced T‐SQL. Performance Optimisation: Experience of monitoring, troubleshooting, and optimising data pipeline and storage performance within ...

experience building and maintaining production pipelines and data models. Deep proficiency with Databricks or similar platforms — Delta Live Tables, Unity Catalog, Delta Lake, and PySpark or Spark SQL as primary tools. Strong SQL skills: complex transformations, incremental materialization, schema evolution, and data quality validation. Solid AWS experience relevant ...

skills (Conceptual, Logical, Physical and Dimensional)Extensive Azure Databricks experience within large-scale enterprise environmentsDatabricks Lakehouse Architecture and Medallion Design PatternsDatabricks Data Engineering using PySpark and Spark SQLDelta Live Tables (DLT)Unity Catalog implementation and governanceDatabricks WorkflowsDatabricks Asset Bundles (DABs)Databricks Solution AcceleratorsMetadata-Driven Ingestion and Orchestration FrameworksMLflow, Feature ...

following: OneLake Fabric Lakehouse Fabric Warehouse Data Factory Notebooks Power BI Semantic Models Dataflows ✅ Advanced SQL development skills ✅ Python and/or PySpark experience ✅ Strong understanding of dimensional modelling and data warehousing concepts ✅ Experience building enterprise-scale data integration and pipeline solutions ✅ Strong Power BI development expertise, including ...

data quality best practice.Strong problem-solving skills and attention to detail.The ability to communicate effectively with both technical and business stakeholders.Desirable ExperiencePython or PySpark development within Microsoft Fabric.CI/CD practices for data engineering and data product delivery.Experience within legal services or another professional services environment.Exposure to systems such ...

Python/Data Engineering professionals with a solid Azure foundation and the ability to learn and work across the wider platform. Desirable Databricks/PySpark NoSQL/Cosmos DB Infrastructure as Code/DevOps automation AI/ML data integration Automotive or insurance industry experience If this sounds ...

have To be considered for this role, we would love you to have: Extensive experience in programming/query language skills including Python/PySpark and SQL. Data engineering experience - via data preparation, transformation & conversion to allow for rapid processing of system data across a number of different formats ...

EXPERIENCE You should have: Previous experience leading or managing Data Engineers Strong consulting or client-facing experience Advanced SQL and Python skills Experience with PySpark Experience with Azure, Microsoft Fabric and Databricks (or strong cloud data engineering experience) CI/CD and Git experience Power BI knowledge Experience building ...

knowledge: Experience of designing and implementing an Azure analytical platform including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Delta Lake, Unity Catalog, PySpark, SQL, Python, and Azure DevOps. Experience of working with senior stakeholders to articulate the strategy, objectives and benefits of a central enterprise cloud data ...

Support Stakeholder Engagement & Collaboration Independent Troubleshooting & Problem Solving Agile/Scrum Development Azure DevOps, CI/CD & Release Management Technical Documentation Technical Azure Databricks, PySpark, Python & SQL Azure Data Factory Azure Functions Azure Data Lake/Lakehouse REST APIs & Third-Party System Integration Azure SQL Logic Apps Azure DevOps ...

contribute to long-term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse ...

working in agile teams Strong judgment with the ability to balance strategic vision with pragmatic, incremental delivery Technical Skills Programming & Analysis: Python, R, SQL, PySpark Cloud Platforms: AWS, Azure, GCP Data Tools: Data profiling tools, data quality platforms, metadata management systems Visualization: Tableau, Power BI, or similar BI tools ...

APIs, Prometheus, or Azure Monitor.• Maintain best practices for data engineering performance, testing, and documentation.Technical Skills Required• Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake.• Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies.• Experience tuning job configurations (e.g., memory management, parallelism ...

analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience with core technologies such as Python, PySpark, SQL, Airflow, Databricks, Kedro, Dask/RAPIDS, Docker, Kubernetes, and cloud services (AWS/GCP/Azure)Experience with Generative AI (GenAI) or agentic systems ...