26 to 50 of 119 Permanent PySpark Jobs

5+ years of experience in one or more of the following statistical/analytic languages such as Python (Pandas, Scikit-Learn), Apache Spark (or PySpark), Hive, and Scala in a cloud computing environment 5+ years of experience in one or more of the following: database query and management tools ...

Engineer to join a key programme. You'll be responsible for building scalable data pipelines and data lake architecture across Azure Databricks , using Python, PySpark, SQL, Azure Data Factory and Microsoft Fabric . We're looking for someone who has previously delivered data engineering projects within the UK public ...

solution design, workshops and Proof of Concepts. What you'll bring: Proven experience designing enterprise data platforms using Databricks . Strong knowledge of Spark, PySpark, Delta Lake, SQL and modern data engineering. Expertise with Microsoft Azure Strong understanding of data modelling, governance and ETL pipelines. Excellent stakeholder management ...

also a key driver in translating Vision and Data Strategy into IT solutions and delivering on them. Strong hands-on expertise in Databricks, PySpark, Python and SQL is central to this role. Key Skills Azure Stack ELT Experience Data Modeling Data Integration & Ingestion Data Manipulation and Processing GitHub, GitHub ...

design and delivery of enterprise-scale Azure Data Platform and Databricks solutions. Strong hands-on expertise with Azure Databricks, Structured Streaming, Delta Live Tables, PySpark, Python and SQL, with a focus on performance optimisation. Strong understanding of cloud data architecture, data governance, data quality and engineering best practices. Excellent ...

years consulting (or equivalent) experience required 7+ years Databricks solution design, implementation, engineering experience required Either 5+ years building data pipelines in Databricks using PySpark, Scala and/or Spark SQL and ability to work across structured, semi-structured and unstructured data OR 5+ years ML Engineering ...

Strong knowledge of data extraction, transformation, and loading (ETL) processes, leveraging tools such as Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi, or scripting languages (Python, PySpark, SQL). Solid understanding of data warehousing and data modelling techniques (Star Schema, Snowflake Schema). Familiarity with security frameworks (GDPR, HIPAA ...

solutions Requirements: Enterprise data warehouse architecture and design Microsoft Fabric, Azure Data Platform technologies and modern Lakehouse architectures Databricks, Unity Catalog and Delta Lake PySpark, Databricks, SQL Medallion Architecture implementation Data modelling approaches including Star Schema and Data Vault Data governance, security and access management Why This Opportunity Greenfield ...

solutions Requirements: Enterprise data warehouse architecture and design Microsoft Fabric, Azure Data Platform technologies and modern Lakehouse architectures Databricks, Unity Catalog and Delta Lake PySpark, Databricks, SQL Medallion Architecture implementation Data modelling approaches including Star Schema and Data Vault Data governance, security and access management Why This Opportunity Greenfield ...

optimisation and leading delivery teams. Strong experience designing and delivering cloud-native data platforms within Azure and/or AWS environments. Advanced experience with PySpark, SparkSQL and large-scale distributed data processing. Strong understanding of modern data architecture, ETL/ELT frameworks and data modelling. Excellent SQL skills ...

reuse. Preferred qualifications, capabilities, and skills Experience with modern data platforms such as Databricks or Snowflake Deep hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks ...

tools, and emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated ...

cloud‐native experience Preferred Qualifications, Capabilities, and Skills Experience with modern data technologies such as Databricks or Snowflake Hands‐on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence, machine ...

cloud‐native experience Preferred Qualifications, Capabilities, and Skills: Experience with modern data technologies such as Databricks or Snowflake Hands‐on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence, machine ...

MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark for working with large‐scale data and machine learning systems. An understanding of feature stores and related data technologies used to support operational machine ...

also a key driver in translating Vision and Data Strategy into IT solutions and delivering on them. Strong hands-on expertise in Databricks, PySpark, Python and SQL is central to this role. Key Skills: Azure Stack ELT Experience Data Modeling Data Integration & Ingestion Data Manipulation and Processing GitHub, GitHub ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation. Experience with Databricks and PySpark for large‐scale data processing. Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring. Experience implementing responsible ...

delivery · Prepare and structure data so that it can be used effectively by AI-enabled analytics and data platform capabilities. · Work with SQL, Python, PySpark, notebooks or platform-native tooling to transform, test and expose data for downstream AI and analytics use cases. · Collaborate with data engineers and architects … understanding of data engineering or analytics engineering concepts, including data modelling, pipelines, ELT/ETL, data quality and analytical data products. · Experience using Python, PySpark, notebooks or equivalent platform tooling for data preparation, analysis, testing or automation. · Understanding of semantic models, business definitions, metrics, metadata and how they improve ...

cloud‐native experience Preferred Qualifications, Capabilities, and Skills Experience with modern data technologies such as Databricks or Snowflake Hands‐on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence, machine ...

complex data led organisation* Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI* Expertise in Streaming Data, Event Driven Design ...

complex data led organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design ...

complex data led organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design ...

solution design, workshops and Proof of Concepts. What you'll bring: Proven experience designing enterprise data platforms using Microsoft Fabric. Strong knowledge of Spark, PySpark, Delta Lake, SQL and modern data engineering. Expertise with Microsoft Azure Excellent stakeholder management and consultancy experience. What's in it for you? 📍Remote ...

Skills – Able to extract meaningful insights from complex datasets and demonstrate clear business impact. Technical Expertise – Strong experience with SQL, Power BI, Python/PySpark; Databricks is a plus but not essential. Stakeholder Management – Confident in engaging with senior stakeholders, translating data into compelling narratives that drive business decisions. ...

communicate future challenges, ensuring operational effectiveness. Skills and Experience: Essential: Full-stack Python Software Developer skills, specialising in numerical processing using Pandas, Numpy, Polars, PySpark, Coroutines, async tasks and Unix Building scalable applications, with intensive risk-based processing. Package software for distribution (eg Docker, Poetry, python-wheels, PyInstaller, Databricks ...