Period
to

The following table provides summary statistics for permanent job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 16 August 2022 with a comparison to the same period in the previous 2 years.

PySpark - Spark Python API
UK
6 months to
16 Aug 2022
Same period 2021 Same period 2020
Rank 695 777 697
Rank change year-on-year +82 -80 +319
Permanent jobs citing PySpark 622 250 80
As % of all permanent jobs advertised in the UK 0.37% 0.22% 0.13%
As % of the Libraries, Frameworks & Software Standards category 0.96% 0.54% 0.32%
Number of salaries quoted 216 150 58
10th Percentile £50,000 £44,900 £51,750
25th Percentile £58,750 £55,313 £60,000
Median annual salary (50th Percentile) £72,500 £66,500 £67,500
Median % change year-on-year +9.02% -1.48% -12.34%
75th Percentile £82,500 £83,750 £80,313
90th Percentile £90,000 £107,500 £103,250
UK excluding London median annual salary £70,000 £57,500 £57,500
% change year-on-year +21.74% - -6.71%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 65,014 46,715 25,363
As % of all permanent IT jobs advertised in the UK 38.96% 40.53% 40.82%
Number of salaries quoted 29,012 30,519 21,187
10th Percentile £38,750 £35,873 £33,750
25th Percentile £47,500 £43,750 £42,500
Median annual salary (50th Percentile) £65,000 £57,500 £55,000
Median % change year-on-year +13.04% +4.55% +4.76%
75th Percentile £83,750 £77,500 £75,000
90th Percentile £100,000 £95,000 £90,500
UK excluding London median annual salary £55,000 £50,000 £50,000
% change year-on-year +10.00% - +11.11%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Salary Trend

3-month moving average salary quoted in jobs citing PySpark.

Salary trend for PySpark in the UK

PySpark
Salary Histogram

Salary distribution for jobs citing PySpark over the 6 months to 16 August 2022.

Salary histogram for PySpark in the UK

PySpark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the UK over the 6 months to 16 August 2022. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +67 534 £72,500 +9.02% 51
London +43 430 £75,000 +7.14% 30
Work from Home -26 287 £70,000 +12.00% 21
UK excluding London -50 105 £70,000 +21.74% 22
North of England -28 53 £80,000 +28.00% 3
Yorkshire -16 34 £65,000 -7.14% 1
Midlands -28 32 £65,000 +33.33% 5
West Midlands -25 30 £65,000 +33.33% 3
North West -43 16 £80,000 +33.33% 1
South East -41 12 £70,000 +42.86% 4
Scotland -34 6 £57,500 - 1
North East - 3 £75,000 - 1
East Midlands -21 2 £65,000 - 2
East of England - 2 - - 5
South West -6 1 £57,500 -8.00% 3

For the 6 months to 16 August 2022, IT jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for PySpark.

1 552 (88.75%) Python
2 393 (63.18%) SQL
3 337 (54.18%) Analytics
4 329 (52.89%) AWS
5 313 (50.32%) Azure
6 306 (49.20%) Data Pipeline
7 302 (48.55%) Apache Spark
8 289 (46.46%) Data Science
9 261 (41.96%) Agile
10 254 (40.84%) ETL
11 253 (40.68%) Databricks
12 249 (40.03%) Data Engineering
13 248 (39.87%) CI/CD
14 245 (39.39%) Data Lake
15 224 (36.01%) Big Data
16 202 (32.48%) Data Warehouse
17 198 (31.83%) Finance
18 189 (30.39%) Scala
19 176 (28.30%) Kafka
20 172 (27.65%) Machine Learning
21 154 (24.76%) Data Modelling
22 150 (24.12%) GCP
23 148 (23.79%) Analytical Skills
23 148 (23.79%) Hadoop
24 147 (23.63%) Business Intelligence
25 143 (22.99%) Social Skills
26 134 (21.54%) Amazon S3
27 127 (20.42%) Java
28 126 (20.26%) Git (software)
28 126 (20.26%) Terraform

PySpark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 302 (48.55%) Apache Spark
2 112 (18.01%) Apache Airflow
3 108 (17.36%) Apache
4 23 (3.70%) SAS
5 19 (3.05%) Luigi
6 12 (1.93%) Apache Pig
6 12 (1.93%) Tornado
7 10 (1.61%) Jupyter
8 4 (0.64%) Confluence
9 3 (0.48%) SharePoint
10 1 (0.16%) Apache Synapse
10 1 (0.16%) CMS
10 1 (0.16%) Pachyderm
Applications
1 10 (1.61%) Microsoft Excel
2 9 (1.45%) Microsoft PowerPoint
Business Applications
1 5 (0.80%) Acoustic Campaign
1 5 (0.80%) Adobe Campaign
1 5 (0.80%) SAP Oil and Gas
2 3 (0.48%) Infor M3
Cloud Services
1 329 (52.89%) AWS
2 313 (50.32%) Azure
3 150 (24.12%) GCP
4 134 (21.54%) Amazon S3
5 111 (17.85%) Azure Data Factory
6 93 (14.95%) Snowflake
7 79 (12.70%) Amazon Kinesis
7 79 (12.70%) Power Platform
8 72 (11.58%) Azure Synapse Analytics
9 68 (10.93%) Serverless
10 57 (9.16%) AWS Lambda
11 54 (8.68%) Azure DevOps
12 33 (5.31%) Amazon EMR
13 26 (4.18%) Azure Cosmos DB
14 24 (3.86%) dbt
15 23 (3.70%) Amazon SQS
16 21 (3.38%) AWS Step Functions
17 18 (2.89%) Azure Monitor
18 17 (2.73%) GitHub
19 16 (2.57%) Datadog
Communications & Networking
1 1 (0.16%) Broadband
1 1 (0.16%) DNS
1 1 (0.16%) Firewall
Database & Business Intelligence
1 245 (39.39%) Data Lake
2 224 (36.01%) Big Data
3 202 (32.48%) Data Warehouse
4 148 (23.79%) Hadoop
5 100 (16.08%) Amazon Redshift
6 92 (14.79%) NoSQL
7 87 (13.99%) Apache Hive
8 79 (12.70%) Tableau
9 78 (12.54%) Power BI
10 75 (12.06%) SQL Server
11 72 (11.58%) Amazon Athena
12 56 (9.00%) Relational Database
13 51 (8.20%) Spark SQL
13 51 (8.20%) Sqoop
14 46 (7.40%) PostgreSQL
15 43 (6.91%) BigQuery
16 42 (6.75%) SQL Server Integration Services
17 36 (5.79%) Electronic Patient Records
18 34 (5.47%) SQL Server Reporting Services
19 33 (5.31%) RDBMS
Development Applications
1 126 (20.26%) Git (software)
2 98 (15.76%) Jenkins
3 62 (9.97%) CircleCI
4 34 (5.47%) JIRA
5 26 (4.18%) GitLab
6 20 (3.22%) Visual Studio
7 17 (2.73%) MLflow
8 14 (2.25%) Atlassian Bamboo
9 12 (1.93%) ER Studio
9 12 (1.93%) ERwin
9 12 (1.93%) PowerDesigner
10 8 (1.29%) SQL Server Management Studio (SSMS)
11 4 (0.64%) Artifactory
12 2 (0.32%) Bitbucket Server
12 2 (0.32%) Gerrit
12 2 (0.32%) IBM UrbanCode
12 2 (0.32%) Maven
13 1 (0.16%) ProC
13 1 (0.16%) Travis CI
General
1 198 (31.83%) Finance
2 148 (23.79%) Analytical Skills
3 143 (22.99%) Social Skills
4 72 (11.58%) Marketing
5 58 (9.32%) Inclusion and Diversity
6 54 (8.68%) Banking
6 54 (8.68%) Retail
7 35 (5.63%) Influencing Skills
8 30 (4.82%) Games
9 22 (3.54%) Public Sector
10 19 (3.05%) Front Office
11 17 (2.73%) Esports
11 17 (2.73%) Online Betting
12 14 (2.25%) Advertising
13 10 (1.61%) Telecoms
14 9 (1.45%) Law
14 9 (1.45%) Legal
14 9 (1.45%) Local Government
14 9 (1.45%) Pharmaceutical
15 5 (0.80%) Presentation Skills
Job Titles
1 409 (65.76%) Data Engineer
2 99 (15.92%) Senior Data Engineer
3 78 (12.54%) Azure Engineer
4 75 (12.06%) Azure Data Engineer
5 64 (10.29%) Data Scientist
6 41 (6.59%) Python Engineer
7 39 (6.27%) Senior Data Scientist
8 38 (6.11%) Developer
9 34 (5.47%) Cloud Engineer
10 26 (4.18%) Python Developer
11 25 (4.02%) Big Data Engineer
12 24 (3.86%) Software Engineer
13 19 (3.05%) Analyst
14 18 (2.89%) AWS Engineer
14 18 (2.89%) Data Analyst
15 17 (2.73%) AWS Data Engineer
16 12 (1.93%) Data Manager
16 12 (1.93%) Lead Cloud Engineer
16 12 (1.93%) Operations Engineer
17 11 (1.77%) Senior Analyst
Libraries, Frameworks & Software Standards
1 176 (28.30%) Kafka
2 72 (11.58%) Pandas
3 63 (10.13%) Spark Streaming
4 38 (6.11%) OAuth
5 34 (5.47%) NumPy
6 32 (5.14%) .NET
7 27 (4.34%) Flask
8 25 (4.02%) TensorFlow
9 19 (3.05%) Apache NiFi
10 17 (2.73%) ARM Templates
10 17 (2.73%) Keras
10 17 (2.73%) RabbitMQ
11 16 (2.57%) YAML
12 15 (2.41%) scikit-learn
13 14 (2.25%) SciPy
14 13 (2.09%) PyTorch
15 12 (1.93%) AngularJS
15 12 (1.93%) Dask
16 10 (1.61%) Apache Avro
17 9 (1.45%) Django
Miscellaneous
1 99 (15.92%) Algorithms
2 80 (12.86%) Data Structures
3 69 (11.09%) Greenfield Project
4 26 (4.18%) IoT
5 25 (4.02%) Self-Motivation
6 24 (3.86%) Cloud Native
6 24 (3.86%) Public Cloud
7 18 (2.89%) Industry 4.0
7 18 (2.89%) User Experience
8 17 (2.73%) Online Games
8 17 (2.73%) Sports Betting
9 16 (2.57%) Equities
10 14 (2.25%) Blog
11 12 (1.93%) Derivative
11 12 (1.93%) PropTech
12 11 (1.77%) Credit Risk
13 8 (1.29%) Analytical Mindset
13 8 (1.29%) Educational Technology
13 8 (1.29%) Management Information System
14 6 (0.96%) Renewable Energy
Operating Systems
1 20 (3.22%) Linux
2 12 (1.93%) CentOS
2 12 (1.93%) Unix
3 7 (1.13%) Windows
4 2 (0.32%) Mac OS X
5 1 (0.16%) MVS
Processes & Methodologies
1 337 (54.18%) Analytics
2 306 (49.20%) Data Pipeline
3 289 (46.46%) Data Science
4 261 (41.96%) Agile
5 254 (40.84%) ETL
6 249 (40.03%) Data Engineering
7 248 (39.87%) CI/CD
8 172 (27.65%) Machine Learning
9 154 (24.76%) Data Modelling
10 147 (23.63%) Business Intelligence
11 117 (18.81%) DevOps
12 108 (17.36%) Software Engineering
13 103 (16.56%) Data Analytics
14 100 (16.08%) SDLC
15 86 (13.83%) Visualisation
16 84 (13.50%) Data Transformation
17 80 (12.86%) Problem-Solving
18 74 (11.90%) Data Analysis
19 71 (11.41%) Artificial Intelligence
20 68 (10.93%) Data Architecture
Programming Languages
1 552 (88.75%) Python
2 393 (63.18%) SQL
3 189 (30.39%) Scala
4 127 (20.42%) Java
5 42 (6.75%) C#
6 40 (6.43%) R
7 36 (5.79%) Bash
8 30 (4.82%) PowerShell
9 23 (3.70%) T-SQL
10 20 (3.22%) TypeScript
11 15 (2.41%) JavaScript
12 12 (1.93%) Go
13 10 (1.61%) Shell Script
14 6 (0.96%) C
14 6 (0.96%) Data Analysis Expressions (DAX)
14 6 (0.96%) SAS Macro
15 4 (0.64%) Kotlin
16 3 (0.48%) PL/SQL
17 2 (0.32%) PHP
17 2 (0.32%) Ruby
Qualifications
1 81 (13.02%) Degree
2 30 (4.82%) PhD
3 25 (4.02%) AWS Certification
4 24 (3.86%) Computer Science Degree
5 17 (2.73%) Master's Degree
6 7 (1.13%) PMI Certification
6 7 (1.13%) SC Cleared
6 7 (1.13%) Security Cleared
7 5 (0.80%) Microsoft Certification
8 3 (0.48%) Numerate Degree
9 2 (0.32%) FFIEC
9 2 (0.32%) PMP
Quality Assurance & Compliance
1 106 (17.04%) Data Quality
2 36 (5.79%) QA
3 2 (0.32%) GLBA
3 2 (0.32%) NIST
3 2 (0.32%) PCI DSS
4 1 (0.16%) GDPR
System Software
1 56 (9.00%) Docker
2 50 (8.04%) Apache Flume
3 4 (0.64%) HDFS
4 2 (0.32%) Active Directory
Systems Management
1 126 (20.26%) Terraform
2 72 (11.58%) Kubernetes
3 39 (6.27%) Oozie
4 14 (2.25%) Ansible
5 6 (0.96%) Argo
6 4 (0.64%) Consul
7 3 (0.48%) Grafana
8 2 (0.32%) Istio
8 2 (0.32%) Opscode Chef
8 2 (0.32%) Puppet
8 2 (0.32%) Rundeck
9 1 (0.16%) Kubeflow
Vendors
1 253 (40.68%) Databricks
2 87 (13.99%) Microsoft
3 38 (6.11%) Cloudera
4 33 (5.31%) Google
5 30 (4.82%) Oracle
6 18 (2.89%) Foundry
6 18 (2.89%) Palantir
6 18 (2.89%) Salesforce.com
7 17 (2.73%) Zopa
8 10 (1.61%) Teradata
9 9 (1.45%) Dataiku
10 8 (1.29%) Informatica
11 7 (1.13%) SAP
12 6 (0.96%) PagerDuty
12 6 (0.96%) Samsung
12 6 (0.96%) Sun
13 5 (0.80%) Adobe
13 5 (0.80%) PayPal
13 5 (0.80%) Salesforce Marketing Cloud
14 4 (0.64%) LexisNexis