Period
to 21 September 2021

The following table provides summary statistics for permanent job vacancies advertised in the City of London with a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 21 September 2021 with a comparison to the same period in the previous 2 years.

PySpark - Spark Python API
Central London > City of London
6 months to
21 Sep 2021
Same period 2020 Same period 2019
Rank 213 186 315
Rank change year-on-year -27 +129 +39
Permanent jobs citing PySpark 38 11 9
As % of all permanent jobs advertised in the City of London 0.64% 0.28% 0.090%
As % of the Libraries, Frameworks & Software Standards category 1.60% 0.65% 0.23%
Number of salaries quoted 37 11 7
10th Percentile £52,200 £57,500 £56,065
25th Percentile £60,000 £65,625 £67,500
Median annual salary (50th Percentile) £80,000 £97,500 £85,000
Median % change year-on-year -17.95% +14.71% +30.77%
75th Percentile £108,750 £105,000 £91,250
90th Percentile £120,000 " £95,000
Central London median annual salary £80,000 £75,750 £85,000
% change year-on-year +5.61% -10.88% +25.93%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies advertised in the City of London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
City of London
Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 2,374 1,688 3,922
As % of all permanent IT jobs advertised in the City of London 40.18% 43.73% 39.10%
Number of salaries quoted 2,156 1,475 3,560
10th Percentile £47,500 £47,500 £44,400
25th Percentile £57,500 £57,500 £55,000
Median annual salary (50th Percentile) £75,000 £70,000 £70,000
Median % change year-on-year +7.14% - +7.69%
75th Percentile £95,000 £86,250 £85,000
90th Percentile £110,000 £102,500 £95,000
Central London median annual salary £75,000 £70,000 £70,000
% change year-on-year +7.14% - +7.69%

PySpark
Job Vacancy Trend in the City of London

Job postings citing PySpark as a proportion of all IT jobs advertised in the City of London.

Job vacancy trend for PySpark in the City of London

PySpark
Salary Trend in the City of London

3-month moving average salary quoted in jobs citing PySpark in the City of London.

Salary trend for PySpark in the City of London

PySpark
Salary Histogram in the City of London

Salary distribution for jobs citing PySpark in the City of London over the 6 months to 21 September 2021.

Salary histogram for PySpark in the City of London

For the 6 months to 21 September 2021, IT jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads across the City of London region with a requirement for PySpark.

1 27 (71.05%) Azure
2 25 (65.79%) Analytics
3 23 (60.53%) SQL
3 23 (60.53%) Business Intelligence
4 22 (57.89%) Data Warehouse
5 21 (55.26%) ETL
6 20 (52.63%) Python
7 19 (50.00%) Data Modelling
7 19 (50.00%) Data Lake
8 13 (34.21%) Data Pipeline
8 13 (34.21%) Data Engineering
9 12 (31.58%) AWS
9 12 (31.58%) Power BI
9 12 (31.58%) Databricks
9 12 (31.58%) Power Platform
10 11 (28.95%) Azure Synapse Analytics
10 11 (28.95%) DevOps
10 11 (28.95%) Agile Software Development
10 11 (28.95%) Star Schema
10 11 (28.95%) Microsoft
11 10 (26.32%) Snowflake Schema
11 10 (26.32%) Data Architecture
12 9 (23.68%) Data Mart
12 9 (23.68%) Translating Business Requirements
12 9 (23.68%) Metadata Management
12 9 (23.68%) Metadata
12 9 (23.68%) Storage Management
13 8 (21.05%) Data Strategy
13 8 (21.05%) Azure SQL Database
13 8 (21.05%) Analytical Skills

PySpark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 7 (18.42%) Apache Spark
2 4 (10.53%) Apache Airflow
3 3 (7.89%) Apache
4 2 (5.26%) Apache Pig
4 2 (5.26%) SAS
Cloud Services
1 27 (71.05%) Azure
2 12 (31.58%) AWS
2 12 (31.58%) Power Platform
3 11 (28.95%) Azure Synapse Analytics
4 6 (15.79%) Azure Data Factory
5 5 (13.16%) GCP
5 5 (13.16%) Serverless
6 4 (10.53%) Amazon S3
6 4 (10.53%) AWS Step Functions
7 3 (7.89%) Azure Functions
7 3 (7.89%) Azure Logic Apps
7 3 (7.89%) Snowflake
8 2 (5.26%) Amazon SQS
8 2 (5.26%) AWS Lambda
8 2 (5.26%) SaaS
9 1 (2.63%) Cloud Functions
9 1 (2.63%) dbt
9 1 (2.63%) Firebase
9 1 (2.63%) Fivetran
9 1 (2.63%) Google App Engine
Database & Business Intelligence
1 22 (57.89%) Data Warehouse
2 19 (50.00%) Data Lake
3 12 (31.58%) Power BI
4 11 (28.95%) Star Schema
5 10 (26.32%) Snowflake Schema
6 9 (23.68%) Data Mart
6 9 (23.68%) Metadata
7 8 (21.05%) Azure SQL Database
8 5 (13.16%) Amazon Redshift
8 5 (13.16%) Delta Lake
8 5 (13.16%) Neo4j
9 4 (10.53%) Amazon Athena
9 4 (10.53%) OLTP
10 3 (7.89%) Tableau
11 2 (5.26%) Azure SQL Data Warehouse
11 2 (5.26%) Big Data
11 2 (5.26%) HDInsight
11 2 (5.26%) Parquet
11 2 (5.26%) Spark SQL
11 2 (5.26%) Sqoop
Development Applications
1 2 (5.26%) ER Studio
1 2 (5.26%) ERwin
1 2 (5.26%) Git (software)
1 2 (5.26%) JIRA
1 2 (5.26%) PowerDesigner
2 1 (2.63%) Bitbucket
2 1 (2.63%) GitLab
General
1 8 (21.05%) Analytical Skills
2 2 (5.26%) Advertising
2 2 (5.26%) Law
2 2 (5.26%) Legal
3 1 (2.63%) Banking
3 1 (2.63%) Finance
3 1 (2.63%) Marketing
Job Titles
1 22 (57.89%) Data Engineer
2 13 (34.21%) Architect
2 13 (34.21%) Data Architect
3 7 (18.42%) Senior Data Engineer
4 6 (15.79%) Azure Engineer
5 4 (10.53%) Azure Architect
5 4 (10.53%) Enterprise Architect
5 4 (10.53%) Enterprise Data Architect
6 2 (5.26%) AWS Engineer
6 2 (5.26%) Business Intelligence Developer
6 2 (5.26%) Developer
6 2 (5.26%) Integration Developer
7 1 (2.63%) Python Engineer
Libraries, Frameworks & Software Standards
1 4 (10.53%) MDX
1 4 (10.53%) Middleware
2 2 (5.26%) JSON
2 2 (5.26%) Kafka
2 2 (5.26%) RabbitMQ
3 1 (2.63%) AngularJS
3 1 (2.63%) Apache Tika
3 1 (2.63%) CoreNLP
3 1 (2.63%) Gensim
3 1 (2.63%) Keras
3 1 (2.63%) Laravel
3 1 (2.63%) Node.js
3 1 (2.63%) Pandas
3 1 (2.63%) PyTorch
3 1 (2.63%) REST
3 1 (2.63%) scikit-learn
3 1 (2.63%) SciPy
3 1 (2.63%) SpaCy
3 1 (2.63%) TensorFlow
Miscellaneous
1 2 (5.26%) Derivative
1 2 (5.26%) Greenfield Project
1 2 (5.26%) Renewable Energy
1 2 (5.26%) User Experience
Operating Systems
1 2 (5.26%) Linux
1 2 (5.26%) Unix
Processes & Methodologies
1 25 (65.79%) Analytics
2 23 (60.53%) Business Intelligence
3 21 (55.26%) ETL
4 19 (50.00%) Data Modelling
5 13 (34.21%) Data Engineering
5 13 (34.21%) Data Pipeline
6 11 (28.95%) Agile Software Development
6 11 (28.95%) DevOps
7 10 (26.32%) Data Architecture
8 9 (23.68%) Metadata Management
8 9 (23.68%) Storage Management
8 9 (23.68%) Translating Business Requirements
9 8 (21.05%) Data Strategy
10 7 (18.42%) Visualisation
11 5 (13.16%) Scrum
12 4 (10.53%) Artificial Intelligence
12 4 (10.53%) Critical Success Factor
12 4 (10.53%) Data Analysis
12 4 (10.53%) Innovative Thinking
12 4 (10.53%) Release Management
Programming Languages
1 23 (60.53%) SQL
2 20 (52.63%) Python
3 5 (13.16%) Cypher
4 4 (10.53%) Data Analysis Expressions (DAX)
4 4 (10.53%) PowerShell
5 3 (7.89%) Scala
6 2 (5.26%) Java
6 2 (5.26%) JavaScript
6 2 (5.26%) R
6 2 (5.26%) Shell Script
7 1 (2.63%) PHP
7 1 (2.63%) TypeScript
Qualifications
1 1 (2.63%) Degree
1 1 (2.63%) PhD
System Software
1 2 (5.26%) Apache Flume
2 1 (2.63%) Docker
Systems Management
1 3 (7.89%) Terraform
2 1 (2.63%) Kubernetes
Vendors
1 12 (31.58%) Databricks
2 11 (28.95%) Microsoft
3 2 (5.26%) Oracle
3 2 (5.26%) SAP
3 2 (5.26%) SnapLogic
3 2 (5.26%) Workday