Period
to 25 September 2021

The following table provides summary statistics for permanent job vacancies advertised in London with a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 25 September 2021 with a comparison to the same period in the previous 2 years.

PySpark - Spark Python API
England > London
6 months to
25 Sep 2021
Same period 2020 Same period 2019
Rank 534 412 680
Rank change year-on-year -122 +268 +86
Permanent jobs citing PySpark 178 70 71
As % of all permanent jobs advertised in London 0.36% 0.35% 0.13%
As % of the Libraries, Frameworks & Software Standards category 0.76% 0.76% 0.33%
Number of salaries quoted 111 51 56
10th Percentile £52,500 £51,250 £63,750
25th Percentile £57,500 £57,500 £67,188
Median annual salary (50th Percentile) £70,000 £67,500 £77,500
Median % change year-on-year +3.70% -12.90% -0.64%
75th Percentile £90,000 £76,250 £90,000
90th Percentile £110,000 £105,000 £110,000
England median annual salary £70,000 £65,000 £70,000
% change year-on-year +7.69% -7.14% -6.67%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies advertised in London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
London
Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 23,294 9,180 21,744
As % of all permanent IT jobs advertised in London 47.51% 45.69% 38.96%
Number of salaries quoted 13,320 7,861 18,262
10th Percentile £46,250 £46,250 £40,000
25th Percentile £56,250 £55,000 £51,250
Median annual salary (50th Percentile) £72,500 £72,000 £67,500
Median % change year-on-year +0.69% +6.67% +3.85%
75th Percentile £91,250 £90,000 £85,000
90th Percentile £110,000 £115,000 £100,000
England median annual salary £60,000 £60,000 £55,000
% change year-on-year - +9.09% +5.77%

PySpark
Job Vacancy Trend in London

Job postings citing PySpark as a proportion of all IT jobs advertised in London.

Job vacancy trend for PySpark in London

PySpark
Salary Trend in London

3-month moving average salary quoted in jobs citing PySpark in London.

Salary trend for PySpark in London

PySpark
Salary Histogram in London

Salary distribution for jobs citing PySpark in London over the 6 months to 25 September 2021.

Salary histogram for PySpark in London

PySpark
Job Locations in London

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the London region over the 6 months to 25 September 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -38 37 £78,750 +3.96% 11
South London -24 8 £60,000 -5.88%
West London -71 6 £70,000 -
PySpark
England

For the 6 months to 25 September 2021, IT jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads across the London region with a requirement for PySpark.

1 156 (87.64%) Python
2 123 (69.10%) Analytics
3 122 (68.54%) SQL
4 82 (46.07%) Azure
4 82 (46.07%) AWS
5 75 (42.13%) Apache Spark
6 69 (38.76%) Data Lake
7 64 (35.96%) Data Science
8 60 (33.71%) Data Engineering
9 57 (32.02%) Data Pipeline
10 55 (30.90%) ETL
11 54 (30.34%) Data Warehouse
11 54 (30.34%) Business Intelligence
11 54 (30.34%) Agile Software Development
12 52 (29.21%) Software Engineering
13 51 (28.65%) Data Modelling
14 47 (26.40%) Machine Learning
15 42 (23.60%) Big Data
16 39 (21.91%) Docker
16 39 (21.91%) Snowflake
17 38 (21.35%) Relational Database
18 37 (20.79%) DevOps
18 37 (20.79%) Hadoop
19 36 (20.22%) Power Platform
19 36 (20.22%) Power BI
20 34 (19.10%) Amazon S3
20 34 (19.10%) Finance
21 33 (18.54%) PostgreSQL
21 33 (18.54%) Visualisation
21 33 (18.54%) R

PySpark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 75 (42.13%) Apache Spark
2 26 (14.61%) Apache
3 16 (8.99%) Apache Airflow
4 12 (6.74%) SAS
5 7 (3.93%) Jupyter
6 6 (3.37%) Elasticsearch
6 6 (3.37%) Oracle SOA Suite
7 2 (1.12%) Apache Pig
7 2 (1.12%) Apache Solr
8 1 (0.56%) Luigi
8 1 (0.56%) Tornado
Applications
1 4 (2.25%) Microsoft Excel
Cloud Services
1 82 (46.07%) AWS
1 82 (46.07%) Azure
2 39 (21.91%) Snowflake
3 36 (20.22%) Power Platform
4 34 (19.10%) Amazon S3
5 33 (18.54%) GCP
6 27 (15.17%) Amazon SQS
7 26 (14.61%) GitHub
8 20 (11.24%) Amazon ECS
9 18 (10.11%) Azure Synapse Analytics
10 13 (7.30%) Azure Data Factory
10 13 (7.30%) Serverless
11 12 (6.74%) Amazon Kinesis
12 11 (6.18%) IaaS
12 11 (6.18%) PaaS
12 11 (6.18%) SaaS
13 8 (4.49%) AWS Lambda
14 7 (3.93%) Amazon SageMaker
14 7 (3.93%) Azure Logic Apps
14 7 (3.93%) dbt
Communications & Networking
1 2 (1.12%) Internet
Database & Business Intelligence
1 69 (38.76%) Data Lake
2 54 (30.34%) Data Warehouse
3 42 (23.60%) Big Data
4 38 (21.35%) Relational Database
5 37 (20.79%) Hadoop
6 36 (20.22%) Power BI
7 33 (18.54%) PostgreSQL
8 28 (15.73%) Amazon RDS
9 24 (13.48%) Tableau
10 23 (12.92%) SQL Server
11 21 (11.80%) NoSQL
12 17 (9.55%) Amazon Redshift
13 16 (8.99%) Apache Hive
14 14 (7.87%) Amazon Athena
14 14 (7.87%) MongoDB
15 13 (7.30%) Azure SQL Database
15 13 (7.30%) Looker
15 13 (7.30%) Spark SQL
15 13 (7.30%) Star Schema
16 12 (6.74%) Flink
Development Applications
1 20 (11.24%) CircleCI
2 19 (10.67%) Git (software)
3 10 (5.62%) JIRA
4 4 (2.25%) Bitbucket
4 4 (2.25%) MLflow
5 3 (1.69%) GitLab
5 3 (1.69%) SonarQube
5 3 (1.69%) TeamCity
6 2 (1.12%) ER Studio
6 2 (1.12%) ERwin
6 2 (1.12%) PowerDesigner
7 1 (0.56%) Jenkins
General
1 34 (19.10%) Finance
2 33 (18.54%) Analytical Skills
3 26 (14.61%) Social Skills
4 17 (9.55%) Retail
5 16 (8.99%) Banking
6 7 (3.93%) Marketing
7 6 (3.37%) Telecoms
8 4 (2.25%) Advertising
8 4 (2.25%) Presentation Skills
9 3 (1.69%) Law
9 3 (1.69%) Manufacturing
10 2 (1.12%) Electronics
10 2 (1.12%) Inclusion and Diversity
10 2 (1.12%) Investment Banking
10 2 (1.12%) Legal
10 2 (1.12%) Pharmaceutical
10 2 (1.12%) Public Sector
11 1 (0.56%) Front Office
Job Titles
1 73 (41.01%) Data Engineer
2 29 (16.29%) Data Scientist
3 28 (15.73%) Architect
4 23 (12.92%) Data Architect
5 19 (10.67%) Senior Data Engineer
6 15 (8.43%) Software Engineer
7 14 (7.87%) Python Engineer
8 10 (5.62%) Azure Architect
8 10 (5.62%) Lead Software Engineer
9 8 (4.49%) Big Data Architect
9 8 (4.49%) Cloud Architect
10 7 (3.93%) Developer
11 6 (3.37%) Azure Engineer
11 6 (3.37%) Hadoop Architect
11 6 (3.37%) Java Architect
12 4 (2.25%) Analyst
12 4 (2.25%) Data Analyst
12 4 (2.25%) Enterprise Architect
12 4 (2.25%) Machine Learning Engineer
12 4 (2.25%) Telecoms Engineer
Libraries, Frameworks & Software Standards
1 24 (13.48%) Django
2 18 (10.11%) Kafka
3 9 (5.06%) MLlib
3 9 (5.06%) SparkR
4 8 (4.49%) MDX
5 7 (3.93%) Pandas
6 5 (2.81%) NumPy
6 5 (2.81%) RabbitMQ
6 5 (2.81%) REST
7 4 (2.25%) .NET
7 4 (2.25%) HTML
7 4 (2.25%) Middleware
7 4 (2.25%) scikit-learn
7 4 (2.25%) TensorFlow
8 3 (1.69%) .NET Core
8 3 (1.69%) Angular 2
8 3 (1.69%) AngularJS
8 3 (1.69%) Apache NiFi
8 3 (1.69%) Dask
9 2 (1.12%) Apache Avro
Miscellaneous
1 20 (11.24%) Algorithms
2 11 (6.18%) Greenfield Project
3 6 (3.37%) Management Information System
3 6 (3.37%) Virtual Team
4 5 (2.81%) Derivative
5 3 (1.69%) Data Structures
5 3 (1.69%) Fixed Income
5 3 (1.69%) Legacy Systems
5 3 (1.69%) Reference Data
5 3 (1.69%) User Experience
6 2 (1.12%) Life Science
6 2 (1.12%) Renewable Energy
6 2 (1.12%) Self-Motivation
6 2 (1.12%) Social Media
7 1 (0.56%) Cloud Native
7 1 (0.56%) Credit Risk
7 1 (0.56%) FMCG
7 1 (0.56%) Foreign Exchange (FX)
7 1 (0.56%) Mobile App
7 1 (0.56%) Taxonomies
Operating Systems
1 18 (10.11%) Unix
2 9 (5.06%) Windows
3 7 (3.93%) Linux
Processes & Methodologies
1 123 (69.10%) Analytics
2 64 (35.96%) Data Science
3 60 (33.71%) Data Engineering
4 57 (32.02%) Data Pipeline
5 55 (30.90%) ETL
6 54 (30.34%) Agile Software Development
6 54 (30.34%) Business Intelligence
7 52 (29.21%) Software Engineering
8 51 (28.65%) Data Modelling
9 47 (26.40%) Machine Learning
10 37 (20.79%) DevOps
11 33 (18.54%) Visualisation
12 30 (16.85%) Microservices
13 29 (16.29%) Mathematics
14 26 (14.61%) Data Architecture
14 26 (14.61%) Mentoring
15 25 (14.04%) Continuous Integration
16 24 (13.48%) Artificial Intelligence
16 24 (13.48%) Dashboard Development
16 24 (13.48%) Data Strategy
Programming Languages
1 156 (87.64%) Python
2 122 (68.54%) SQL
3 33 (18.54%) R
4 30 (16.85%) Java
5 25 (14.04%) Scala
6 13 (7.30%) Shell Script
7 8 (4.49%) Data Analysis Expressions (DAX)
7 8 (4.49%) T-SQL
8 7 (3.93%) Cypher
8 7 (3.93%) JavaScript
8 7 (3.93%) PL/SQL
9 6 (3.37%) C#
9 6 (3.37%) PowerShell
10 3 (1.69%) HiveQL
10 3 (1.69%) LookML
10 3 (1.69%) TypeScript
11 2 (1.12%) Bash Shell
11 2 (1.12%) C++
11 2 (1.12%) SPARQL
Qualifications
1 28 (15.73%) Degree
2 10 (5.62%) PhD
3 2 (1.12%) AWS Certification
3 2 (1.12%) BPSS Clearance
3 2 (1.12%) SC Cleared
3 2 (1.12%) Security Cleared
4 1 (0.56%) Computer Science Degree
4 1 (0.56%) MBA
4 1 (0.56%) Microsoft Certification
Quality Assurance & Compliance
1 9 (5.06%) Data Quality
2 4 (2.25%) QA
System Software
1 39 (21.91%) Docker
2 5 (2.81%) HDFS
3 3 (1.69%) Apache ZooKeeper
4 2 (1.12%) Apache Flume
5 1 (0.56%) Hyper-V
5 1 (0.56%) Virtual Machines
Systems Management
1 26 (14.61%) Kubernetes
2 11 (6.18%) Terraform
3 7 (3.93%) Oozie
4 3 (1.69%) Hadoop YARN
5 1 (0.56%) Ansible
5 1 (0.56%) Control-M
5 1 (0.56%) Mesos
Vendors
1 33 (18.54%) Databricks
2 28 (15.73%) Microsoft
3 19 (10.67%) Google
4 15 (8.43%) Oracle
5 9 (5.06%) Cloudera
5 9 (5.06%) Hortonworks
6 6 (3.37%) Informatica
7 4 (2.25%) SAP
7 4 (2.25%) Sybase
8 3 (1.69%) Foundry
8 3 (1.69%) Palantir
8 3 (1.69%) ServiceNow
9 2 (1.12%) Dataiku
9 2 (1.12%) SnapLogic
9 2 (1.12%) StreamSets
9 2 (1.12%) Workday
10 1 (0.56%) Cadence
10 1 (0.56%) Splunk
10 1 (0.56%) Talend
10 1 (0.56%) VMware