Period
to

The table below provides summary statistics for permanent job vacancies with an option to work from home (WFH) and a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 19 August 2022 with a comparison to the same period in the previous 2 years.

PySpark - Spark Python API
UK > Work from Home
6 months to
19 Aug 2022
Same period 2021 Same period 2020
Rank 587 580 311
Rank change year-on-year -7 -269 +39
Permanent jobs citing PySpark 279 83 2
As % of all permanent jobs with a WFH option 0.37% 0.20% 0.023%
As % of the Libraries, Frameworks & Software Standards category 0.86% 0.42% 0.043%
Number of salaries quoted 85 58 1
10th Percentile £50,000 £43,925 -
25th Percentile £52,500 £51,250 -
Median annual salary (50th Percentile) £70,000 £62,500 £65,000
Median % change year-on-year +12.00% -3.85% -27.78%
75th Percentile £82,500 £82,813 -
90th Percentile £90,000 £107,500 -
UK median annual salary £72,500 £66,500 £67,500
% change year-on-year +9.02% -1.48% -6.90%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a WFH option and a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
Work from Home
Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 32,430 19,873 4,640
As % of all permanent jobs with a WFH option 43.51% 47.89% 52.79%
Number of salaries quoted 17,187 14,444 4,109
10th Percentile £40,000 £35,000 £35,000
25th Percentile £47,500 £42,500 £42,500
Median annual salary (50th Percentile) £65,000 £55,000 £55,000
Median % change year-on-year +18.18% - +4.76%
75th Percentile £82,500 £75,000 £72,500
90th Percentile £95,000 £90,000 £85,000
UK median annual salary £65,000 £57,500 £55,000
% change year-on-year +13.04% +4.55% +4.76%

PySpark
Trend for Jobs with a WFH Option

Job vacancies with a work from home option citing PySpark as a proportion of all IT jobs advertised.

PySpark trend for jobs with a WFH option

PySpark
Salary Trend for Jobs with a WFH Option

3-month moving average salary quoted in job vacancies with a work from home option citing PySpark.

PySpark salary trend for jobs with a WFH option

PySpark
Salary Histogram for Jobs with a WFH Option

Salary distribution for jobs with a work from home option citing PySpark over the 6 months to 19 August 2022.

PySpark salary histogram for jobs with a WFH option

For the 6 months to 19 August 2022, PySpark job roles required the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent jobs with a WFH option and a requirement for PySpark.

1 227 (81.36%) Python
2 185 (66.31%) Data Pipeline
3 180 (64.52%) SQL
4 158 (56.63%) Analytics
5 155 (55.56%) AWS
6 149 (53.41%) Apache Spark
7 148 (53.05%) Azure
8 139 (49.82%) Data Lake
9 137 (49.10%) Data Engineering
10 130 (46.59%) Databricks
11 129 (46.24%) CI/CD
12 127 (45.52%) ETL
13 116 (41.58%) Agile
14 108 (38.71%) Kafka
15 105 (37.63%) Data Warehouse
16 101 (36.20%) Data Science
17 93 (33.33%) Git (software)
18 92 (32.97%) Big Data
18 92 (32.97%) Finance
19 86 (30.82%) Amazon S3
20 83 (29.75%) GCP
21 80 (28.67%) Scala
22 77 (27.60%) Analytical Skills
22 77 (27.60%) Terraform
23 74 (26.52%) Hadoop
24 72 (25.81%) Amazon Redshift
25 69 (24.73%) Data Structures
26 66 (23.66%) Business Intelligence
27 65 (23.30%) Apache Airflow
28 63 (22.58%) Greenfield Project

PySpark
Co-occurring IT Skills with a WFH Option by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 149 (53.41%) Apache Spark
2 65 (23.30%) Apache Airflow
3 49 (17.56%) Apache
4 13 (4.66%) Luigi
5 12 (4.30%) Tornado
6 6 (2.15%) Apache Pig
6 6 (2.15%) SAS
7 2 (0.72%) Confluence
7 2 (0.72%) SharePoint
8 1 (0.36%) Apache Synapse
8 1 (0.36%) Jupyter
Applications
1 4 (1.43%) Microsoft Excel
1 4 (1.43%) Microsoft PowerPoint
Business Applications
1 5 (1.79%) Acoustic Campaign
1 5 (1.79%) Adobe Campaign
2 2 (0.72%) Infor M3
Cloud Services
1 155 (55.56%) AWS
2 148 (53.05%) Azure
3 86 (30.82%) Amazon S3
4 83 (29.75%) GCP
5 55 (19.71%) Azure Data Factory
6 53 (19.00%) Amazon Kinesis
7 51 (18.28%) Snowflake
8 48 (17.20%) Power Platform
9 33 (11.83%) Serverless
10 31 (11.11%) AWS Lambda
11 29 (10.39%) Azure Synapse Analytics
12 28 (10.04%) Azure DevOps
13 19 (6.81%) Amazon EMR
14 17 (6.09%) Azure Cosmos DB
15 16 (5.73%) Azure Monitor
16 15 (5.38%) Amazon CloudWatch
17 11 (3.94%) AWS Step Functions
18 10 (3.58%) Amazon EC2
18 10 (3.58%) AWS Data Pipeline
18 10 (3.58%) dbt
Communications & Networking
1 1 (0.36%) Broadband
1 1 (0.36%) DNS
1 1 (0.36%) Firewall
Database & Business Intelligence
1 139 (49.82%) Data Lake
2 105 (37.63%) Data Warehouse
3 92 (32.97%) Big Data
4 74 (26.52%) Hadoop
5 72 (25.81%) Amazon Redshift
6 57 (20.43%) Amazon Athena
7 56 (20.07%) Apache Hive
8 53 (19.00%) NoSQL
9 47 (16.85%) Power BI
10 46 (16.49%) Tableau
11 41 (14.70%) Sqoop
12 36 (12.90%) BigQuery
13 35 (12.54%) Relational Database
14 33 (11.83%) SQL Server
15 31 (11.11%) RDBMS
16 26 (9.32%) Spark SQL
17 23 (8.24%) SQL Server Reporting Services
18 22 (7.89%) PostgreSQL
19 21 (7.53%) Electronic Patient Records
20 18 (6.45%) MySQL
Development Applications
1 93 (33.33%) Git (software)
2 50 (17.92%) Jenkins
3 36 (12.90%) CircleCI
4 19 (6.81%) Visual Studio
5 8 (2.87%) JIRA
6 6 (2.15%) ER Studio
6 6 (2.15%) ERwin
6 6 (2.15%) PowerDesigner
7 4 (1.43%) MLflow
8 3 (1.08%) SQL Server Management Studio (SSMS)
9 2 (0.72%) IBM UrbanCode
9 2 (0.72%) Maven
10 1 (0.36%) GitLab
10 1 (0.36%) ProC
10 1 (0.36%) Travis CI
General
1 92 (32.97%) Finance
2 77 (27.60%) Analytical Skills
3 50 (17.92%) Banking
4 48 (17.20%) Social Skills
5 43 (15.41%) Inclusion and Diversity
6 40 (14.34%) Retail
7 39 (13.98%) Marketing
8 22 (7.89%) Public Sector
9 20 (7.17%) Games
10 16 (5.73%) Esports
10 16 (5.73%) Online Betting
11 15 (5.38%) Influencing Skills
12 9 (3.23%) Legal
12 9 (3.23%) Local Government
13 5 (1.79%) Advertising
13 5 (1.79%) Police
14 4 (1.43%) Law
15 2 (0.72%) Pharmaceutical
15 2 (0.72%) Presentation Skills
Job Titles
1 201 (72.04%) Data Engineer
2 39 (13.98%) Senior Data Engineer
3 31 (11.11%) Azure Data Engineer
3 31 (11.11%) Azure Engineer
4 24 (8.60%) Developer
5 20 (7.17%) Cloud Engineer
6 18 (6.45%) Python Developer
7 12 (4.30%) Lead Cloud Engineer
8 11 (3.94%) Analyst
8 11 (3.94%) Data Analyst
8 11 (3.94%) Software Engineer
9 9 (3.23%) Data Manager
9 9 (3.23%) Engineering Manager
9 9 (3.23%) Senior Manager
10 8 (2.87%) AWS Engineer
11 7 (2.51%) AWS Data Engineer
11 7 (2.51%) Senior Software Engineer
12 6 (2.15%) Automation Engineer
12 6 (2.15%) Python Engineer
13 4 (1.43%) Senior Data Analyst
Libraries, Frameworks & Software Standards
1 108 (38.71%) Kafka
2 54 (19.35%) Spark Streaming
3 35 (12.54%) OAuth
4 33 (11.83%) Pandas
5 30 (10.75%) .NET
6 16 (5.73%) ARM Templates
6 16 (5.73%) RabbitMQ
7 13 (4.66%) Flask
8 12 (4.30%) AngularJS
9 10 (3.58%) Apache NiFi
10 9 (3.23%) Apache Avro
10 9 (3.23%) NumPy
10 9 (3.23%) Web Services
11 8 (2.87%) Express
12 7 (2.51%) FastAPI
13 6 (2.15%) amCharts
13 6 (2.15%) React
13 6 (2.15%) SciPy
13 6 (2.15%) SQLAlchemy
14 5 (1.79%) RESTful
Miscellaneous
1 69 (24.73%) Data Structures
2 63 (22.58%) Greenfield Project
3 55 (19.71%) Algorithms
4 25 (8.96%) IoT
5 16 (5.73%) Cloud Native
5 16 (5.73%) Online Games
5 16 (5.73%) Sports Betting
6 12 (4.30%) User Experience
7 9 (3.23%) Public Cloud
8 8 (2.87%) Analytical Mindset
9 6 (2.15%) Derivative
9 6 (2.15%) e-Learning
10 5 (1.79%) Acoustics
11 3 (1.08%) FMCG
12 2 (0.72%) Biotechnology
12 2 (0.72%) Housing Association
12 2 (0.72%) Hybrid Cloud
12 2 (0.72%) Self-Motivation
12 2 (0.72%) Social Media
13 1 (0.36%) Virtual Team
Operating Systems
1 4 (1.43%) Linux
2 2 (0.72%) Mac OS X
2 2 (0.72%) Windows
3 1 (0.36%) MVS
3 1 (0.36%) Unix
Processes & Methodologies
1 185 (66.31%) Data Pipeline
2 158 (56.63%) Analytics
3 137 (49.10%) Data Engineering
4 129 (46.24%) CI/CD
5 127 (45.52%) ETL
6 116 (41.58%) Agile
7 101 (36.20%) Data Science
8 66 (23.66%) Business Intelligence
9 63 (22.58%) Software Engineering
10 62 (22.22%) Data Analytics
11 59 (21.15%) SDLC
12 55 (19.71%) Data Transformation
13 53 (19.00%) Problem-Solving
14 51 (18.28%) Machine Learning
15 48 (17.20%) Data Modelling
16 42 (15.05%) Artificial Intelligence
16 42 (15.05%) DevOps
17 41 (14.70%) Asset Management
18 38 (13.62%) Containerisation
19 35 (12.54%) Continuous Delivery
Programming Languages
1 227 (81.36%) Python
2 180 (64.52%) SQL
3 80 (28.67%) Scala
4 57 (20.43%) Java
5 28 (10.04%) C#
6 17 (6.09%) PowerShell
7 14 (5.02%) R
8 10 (3.58%) Bash
9 8 (2.87%) TypeScript
10 4 (1.43%) Kotlin
10 4 (1.43%) T-SQL
11 3 (1.08%) Data Analysis Expressions (DAX)
12 2 (0.72%) JavaScript
12 2 (0.72%) PHP
12 2 (0.72%) Ruby
12 2 (0.72%) SAS Macro
13 1 (0.36%) C++
Qualifications
1 38 (13.62%) Degree
2 15 (5.38%) AWS Certification
2 15 (5.38%) AWS Certified Developer
2 15 (5.38%) AWS Certified Solutions Architect
3 14 (5.02%) Computer Science Degree
4 6 (2.15%) Master's Degree
5 5 (1.79%) PhD
6 2 (0.72%) Azure Certification
6 2 (0.72%) Numerate Degree
Quality Assurance & Compliance
1 57 (20.43%) Data Quality
2 27 (9.68%) QA
System Software
1 41 (14.70%) Apache Flume
2 27 (9.68%) Docker
3 3 (1.08%) HDFS
4 1 (0.36%) Active Directory
Systems Management
1 77 (27.60%) Terraform
2 35 (12.54%) Oozie
3 21 (7.53%) Kubernetes
4 12 (4.30%) Ansible
5 4 (1.43%) Consul
6 2 (0.72%) Istio
7 1 (0.36%) Argo
Vendors
1 130 (46.59%) Databricks
2 45 (16.13%) Microsoft
3 35 (12.54%) Cloudera
4 15 (5.38%) Zopa
5 14 (5.02%) Google
5 14 (5.02%) Oracle
6 11 (3.94%) Salesforce.com
7 6 (2.15%) Foundry
7 6 (2.15%) Palantir
7 6 (2.15%) Sun
8 5 (1.79%) Adobe
8 5 (1.79%) Salesforce Marketing Cloud
9 3 (1.08%) Civica
9 3 (1.08%) Teradata
10 2 (0.72%) Infor
11 1 (0.36%) Informatica
11 1 (0.36%) StreamSets