1 to 25 of 195 PySpark Jobs in London

optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow Experience with containerisation ...

Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell) Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software ...

models in data warehouses, lakehouse environments or analytics tools such as Power BI. Familiarity with modern data engineering frameworks and tools (e.g. Spark/PySpark, Pandas, orchestration tools). Experience with cloud data platforms on AWS or Azure (e.g. Databricks, Redshift, Microsoft Fabric). Understanding of software engineering fundamentals ...

construction, programme controls, PMO, or consultancy environments. Experience with cloud-native architectures, DevOps, CI/CD pipelines, and DataOps practices. Knowledge of Python, Spark, PySpark, Databricks, or equivalent data engineering technologies. Information Management in a complex construction or infrastructure project/programme Experience developing enterprise data products and supporting ...

Good understanding of modern software platform architecture and the role of the data platform within the broader platform ecosystem Good Scala skills (or strong PySpark plus a genuine willingness to work primarily in Scala) Strong SQL skills for transformations, data modelling, and debugging data issues Experience building and supporting ...

/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft Purview. ...

Quality, Data Integration, Data Analytics, Data Visualisation, Data Science and Business Intelligence. Confident in delivering solutions using Databricks & developing and optimising data pipelines using PySpark within Databricks environments. Proven experience building, publishing, and supporting dashboards and reports using Power BI Strong experience and programming skills in languages such ...

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

complex data workflows Excellent communication and collaboration skills in agile, cross-functional teams Ability to mentor junior engineers Nice to have: Experience with Databricks, PySpark, and Delta Lake Условия Competitive salary based on location, experience, and skills Comprehensive benefits package for employees and their families Continuous learning, structured development ...

Azure Data Factory, Azure Databricks, MS Fabric ETL/ELT processes and designing, building and testing data pipelines Building data transformations using Python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

pipelines, stored procedures and reporting solutions to cloud-based platforms. Exposure to Azure DevOps, Git and CI/CD practices. Experience with Python, PySpark, Power BI or real estate technology platforms would be advantageous. Previous exposure to commercial real estate, investment management or alternative asset management environments is highly ...

data engineering or integration environments. 8–10 years’ experience in data engineering and integration , ideally in enterprise‐scale environments. Hands‐on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing ...

data-science role, including delivery of reliable pipelines, reusable data assets and useful analytical models in Microsoft Fabric and Databricks. Advanced Python, SQL and PySpark, including experience with data transformation, analysis and common machine-learning libraries. Hands-on experience designing and operating scalable data pipelines, lakehouse data models, Delta ...

following technical expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both ...

Experience leading or mentoring teams of engineers to deliver high‐quality scalable data solutions. Technical Expertise Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow). Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent. Strong ...

Skills Significant experience leading the design, development, and operation of enterprise‐scale data engineering platforms and teams. Strong technical expertise in SQL and Python, PySpark, or equivalent technologies for data ingestion, transformation, and validation. Experience with Microsoft Fabric or similar modern cloud‐based analytics and data platforms. Strong experience ...

/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft Purview. ...

Looking ForProven hands-on experience with the Databricks platform, including orchestration and enterprise-scale Azure data engineering solutions. Strong technical skills in Python, PySpark, and SQL, with experience working with distributed data processing frameworks. Expertise delivering end-to-end data pipelines across ingestion, transformation, and serving layers within modern ...

analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience with core technologies such as Python, PySpark, SQL, Airflow, Databricks, Kedro, Dask/RAPIDS, Docker, Kubernetes, and cloud services (AWS/GCP/Azure)Experience with Generative AI (GenAI) or agentic systems ...

storage Proficiency with Terraform (ideally the Databricks provider) and Gitbased CI/CD workflows Solid programming skills in Python and SQL; comfortable with PySpark and shell scripting Experience tuning Spark jobs and Delta tables (partitioning, Working knowledge of SSO/SCIM, identity providers and secretsmanagement Understanding of data security ...

best practices, including DevOps, DataOps, and MLOps, will be advantageousWhile we advocate using the right tech for the right task, we often leverage: Python, PySpark, the PyData stack, SQL, Airflow, Databricks, Kedro (our open-source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions such ...

Data Stack, critically ADF and Synapse (experience with Microsoft Fabric is a plus) Highly developed python and data pipeline development knowledge, must include substantial PySpark experience Demonstrable DevOps and DataOps experience with an understanding of best practices for engineering, test and ongoing service delivery An understanding of Infrastructure ...

Need to Be Successful Deep, hands‐on experience building and running production data platforms and products A good level of experience in Python (particularly pyspark) for building data pipelines Strong AWS and Terraform experience, with infrastructure‐as‐code as your default way of working rather than something ...

Data Stack, critically ADF and Synapse (experience with Microsoft Fabric is a plus) Highly developed python and data pipeline development knowledge, must include substantial PySpark experience Demonstrable DevOps and DataOps experience with an understanding of best practices for engineering, test and ongoing service delivery An understanding of Infrastructure ...

Star Schema concepts (Fact and Dimension tables), from the ground up. Git: You're fluent in Git workflows — branching, troubleshooting, code reviews. Python/PySpark/SQL: You write clean, efficient code across all three, and you understand how it performs at scale. Database management: Experience with SQL/ ...