Period
to

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 19 May 2022 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
England > London
6 months to
19 May 2022
Same period 2021 Same period 2020
Rank 383 336 294
Rank change year-on-year -47 -42 +101
Contract jobs citing PySpark 212 139 133
As % of all contract jobs advertised in London 0.55% 0.58% 0.66%
As % of the Libraries, Frameworks & Software Standards category 1.76% 1.58% 1.75%
Number of daily rates quoted 151 75 75
10th Percentile £475 £413 £410
25th Percentile £525 £463 £441
Median daily rate (50th Percentile) £610 £550 £525
Median % change year-on-year +10.91% +4.76% -
75th Percentile £725 £588 £613
90th Percentile £825 £663 £670
England median daily rate £600 £550 £525
% change year-on-year +9.09% +4.76% -
Number of hourly rates quoted 1 0 0
Median hourly rate £78.75 - -
England median hourly rate £78.75 £25.25 -
% change year-on-year +211.88% - -

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
London
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 12,047 8,776 7,592
As % of all contract IT jobs advertised in London 31.16% 36.55% 37.65%
Number of daily rates quoted 9,353 6,572 5,671
10th Percentile £425 £381 £375
25th Percentile £506 £450 £450
Median daily rate (50th Percentile) £600 £525 £525
Median % change year-on-year +14.29% - -
75th Percentile £712 £638 £625
90th Percentile £800 £712 £700
England median daily rate £570 £500 £500
% change year-on-year +14.00% - -
Number of hourly rates quoted 64 35 33
10th Percentile £27.75 £21.27 £23.80
25th Percentile £35.00 £36.50 £31.25
Median hourly rate £45.25 £50.00 £48.85
Median % change year-on-year -9.50% +2.35% +10.57%
75th Percentile £65.44 £65.00 £63.00
90th Percentile £75.71 £68.00 £68.71
England median hourly rate £50.00 £46.00 £46.21
% change year-on-year +8.70% -0.45% +2.88%

PySpark
Job Vacancy Trend in London

Job postings citing PySpark as a proportion of all IT jobs advertised in London.

Job vacancy trend for PySpark in London

PySpark
Contractor Daily Rate Trend in London

3-month moving average daily rate quoted in jobs citing PySpark in London.

Daily rate trend for PySpark in London

PySpark
Daily Rate Histogram in London

Daily rate distribution for jobs citing PySpark in London over the 6 months to 19 May 2022.

Daily rate histogram for PySpark in London

PySpark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the London region over the 6 months to 19 May 2022. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
Central London -37 23 £590 +6.31% 5
South London -11 2 £600 -4.00%
West London - 1 £400 - 2
PySpark
England

For the 6 months to 19 May 2022, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for PySpark.

1 176 (83.02%) Python
2 101 (47.64%) SQL
3 91 (42.92%) Azure
4 88 (41.51%) AWS
5 72 (33.96%) CI/CD
6 70 (33.02%) Agile
7 61 (28.77%) Apache Spark
8 60 (28.30%) DevOps
9 55 (25.94%) Databricks
10 49 (23.11%) Analytics
11 48 (22.64%) Scala
11 48 (22.64%) ETL
12 45 (21.23%) Finance
13 44 (20.75%) Git (software)
14 43 (20.28%) GCP
14 43 (20.28%) Data Pipeline
15 42 (19.81%) Kubernetes
16 41 (19.34%) Banking
17 40 (18.87%) Data Engineering
18 39 (18.40%) Docker
19 38 (17.92%) Test Automation
20 36 (16.98%) Big Data
20 36 (16.98%) Data Modelling
21 34 (16.04%) Java
22 32 (15.09%) Amazon S3
22 32 (15.09%) Pandas
23 31 (14.62%) Social Skills
24 30 (14.15%) Serverless
24 30 (14.15%) Microsoft
25 28 (13.21%) Amazon Redshift

PySpark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 61 (28.77%) Apache Spark
2 22 (10.38%) Apache Airflow
3 12 (5.66%) Apache
4 10 (4.72%) Confluence
5 9 (4.25%) nginx
6 5 (2.36%) Apache Solr
7 4 (1.89%) uWSGI
8 3 (1.42%) Jupyter
9 2 (0.94%) SAS
10 1 (0.47%) Cloud Foundry
10 1 (0.47%) Elasticsearch
Business Applications
1 4 (1.89%) Oracle AML
Cloud Services
1 91 (42.92%) Azure
2 88 (41.51%) AWS
3 43 (20.28%) GCP
4 32 (15.09%) Amazon S3
5 30 (14.15%) Serverless
6 24 (11.32%) AWS Lambda
7 17 (8.02%) Power Platform
8 15 (7.08%) Amazon EMR
8 15 (7.08%) Azure Data Factory
9 11 (5.19%) Amazon EC2
10 8 (3.77%) Azure Synapse Analytics
11 7 (3.30%) Snowflake
12 6 (2.83%) Amazon Kinesis
12 6 (2.83%) GitHub
12 6 (2.83%) Google Compute Engine
13 5 (2.36%) Azure Functions
13 5 (2.36%) Azure Storage
14 4 (1.89%) Amazon SageMaker
14 4 (1.89%) Amazon SQS
14 4 (1.89%) Virtual Private Cloud
Communications & Networking
1 4 (1.89%) SSH
2 2 (0.94%) DHCP
2 2 (0.94%) DNS
2 2 (0.94%) iSCSI
2 2 (0.94%) Kerberos
2 2 (0.94%) TCP/IP
3 1 (0.47%) Internet
3 1 (0.47%) Network Security
Database & Business Intelligence
1 36 (16.98%) Big Data
2 28 (13.21%) Amazon Redshift
3 26 (12.26%) Data Lake
4 20 (9.43%) Data Warehouse
5 19 (8.96%) Hadoop
6 18 (8.49%) PostgreSQL
7 17 (8.02%) Power BI
8 15 (7.08%) Amazon Athena
9 14 (6.60%) NoSQL
10 13 (6.13%) Tableau
11 12 (5.66%) BigQuery
12 11 (5.19%) Apache Hive
12 11 (5.19%) SQL Server
13 8 (3.77%) MySQL
14 7 (3.30%) Amazon RDS
14 7 (3.30%) Apache Cassandra
14 7 (3.30%) Spark SQL
15 6 (2.83%) Azure SQL Database
16 5 (2.36%) Microsoft Analysis Services
16 5 (2.36%) Relational Database
Development Applications
1 44 (20.75%) Git (software)
2 17 (8.02%) Jenkins
3 14 (6.60%) JIRA
4 7 (3.30%) MLflow
5 6 (2.83%) ER Studio
6 4 (1.89%) GitLab
7 3 (1.42%) CircleCI
7 3 (1.42%) Maven
8 2 (0.94%) ERwin
8 2 (0.94%) Gradle
8 2 (0.94%) Hudson
8 2 (0.94%) PyCharm
8 2 (0.94%) SonarQube
8 2 (0.94%) Spinnaker
8 2 (0.94%) TeamCity
9 1 (0.47%) Subversion
9 1 (0.47%) Terragrunt
General
1 45 (21.23%) Finance
2 41 (19.34%) Banking
3 31 (14.62%) Social Skills
4 20 (9.43%) Investment Banking
5 14 (6.60%) Analytical Skills
6 12 (5.66%) Retail
7 9 (4.25%) Front Office
8 5 (2.36%) Financial Institution
8 5 (2.36%) Legal
8 5 (2.36%) Marketing
9 4 (1.89%) German Language
9 4 (1.89%) Law
9 4 (1.89%) Pharmaceutical
9 4 (1.89%) Public Sector
10 3 (1.42%) Documentation Skills
11 2 (0.94%) Presentation Skills
12 1 (0.47%) Inclusion and Diversity
Job Titles
1 103 (48.58%) Data Engineer
2 52 (24.53%) Developer
3 42 (19.81%) Python Developer
4 25 (11.79%) Python Engineer
5 22 (10.38%) AWS Engineer
6 17 (8.02%) Backend Developer
7 16 (7.55%) Azure Engineer
8 13 (6.13%) Senior Data Engineer
9 11 (5.19%) Data Scientist
10 7 (3.30%) Big Data Engineer
11 6 (2.83%) AWS Developer
11 6 (2.83%) Cloud Engineer
11 6 (2.83%) Software Engineer
11 6 (2.83%) Technical Leader
12 5 (2.36%) Architect
12 5 (2.36%) Consultant
12 5 (2.36%) Full Stack Developer
13 4 (1.89%) Analyst
13 4 (1.89%) Data Architect
13 4 (1.89%) Front Office Developer
Libraries, Frameworks & Software Standards
1 32 (15.09%) Pandas
2 28 (13.21%) Kafka
3 26 (12.26%) Django
4 23 (10.85%) RESTful
5 22 (10.38%) Flask
6 21 (9.91%) FastAPI
7 17 (8.02%) Dask
8 14 (6.60%) React
9 13 (6.13%) Bottle
10 12 (5.66%) CSS
11 9 (4.25%) HTML
12 8 (3.77%) GraphQL
12 8 (3.77%) Spark Streaming
13 7 (3.30%) REST
13 7 (3.30%) Web Services
14 6 (2.83%) NumPy
14 6 (2.83%) Spring
14 6 (2.83%) Spring Boot
15 5 (2.36%) HTML5
15 5 (2.36%) Ruby on Rails
Miscellaneous
1 18 (8.49%) Algorithms
2 10 (4.72%) Data Structures
3 7 (3.30%) iPad
4 6 (2.83%) Life Science
5 5 (2.36%) Hedge funds
5 5 (2.36%) Private Cloud
6 4 (1.89%) Data Protection Act
6 4 (1.89%) Greenfield Project
7 3 (1.42%) Cloud Native
7 3 (1.42%) Credit Risk
7 3 (1.42%) FMCG
8 2 (0.94%) Enterprise Storage
8 2 (0.94%) Legacy Systems
8 2 (0.94%) Public Cloud
8 2 (0.94%) Renewable Energy
8 2 (0.94%) Replication
9 1 (0.47%) Blockchain
9 1 (0.47%) Linux Command Line
9 1 (0.47%) Social Media
Operating Systems
1 18 (8.49%) Linux
2 4 (1.89%) Unix
3 3 (1.42%) Ubuntu
4 2 (0.94%) Oracle Linux
4 2 (0.94%) Red Hat Enterprise Linux
4 2 (0.94%) Windows
4 2 (0.94%) Windows Server
4 2 (0.94%) Windows Server 2016
4 2 (0.94%) Windows Server 2019
Processes & Methodologies
1 72 (33.96%) CI/CD
2 70 (33.02%) Agile
3 60 (28.30%) DevOps
4 49 (23.11%) Analytics
5 48 (22.64%) ETL
6 43 (20.28%) Data Pipeline
7 40 (18.87%) Data Engineering
8 38 (17.92%) Test Automation
9 36 (16.98%) Data Modelling
10 27 (12.74%) Data Science
10 27 (12.74%) Machine Learning
11 26 (12.26%) Scrum
12 24 (11.32%) Software Engineering
13 21 (9.91%) Business Intelligence
14 18 (8.49%) API Development
14 18 (8.49%) Problem-Solving
15 17 (8.02%) API Design
15 17 (8.02%) Front End Development
15 17 (8.02%) Microservices
15 17 (8.02%) Observability
Programming Languages
1 176 (83.02%) Python
2 101 (47.64%) SQL
3 48 (22.64%) Scala
4 34 (16.04%) Java
5 15 (7.08%) R
6 7 (3.30%) TypeScript
7 6 (2.83%) C#
8 5 (2.36%) JavaScript
8 5 (2.36%) PHP
8 5 (2.36%) Ruby
9 4 (1.89%) Bash
9 4 (1.89%) Data Analysis Expressions (DAX)
9 4 (1.89%) T-SQL
10 2 (0.94%) PowerShell
11 1 (0.47%) C++
11 1 (0.47%) Cython
11 1 (0.47%) Julia
11 1 (0.47%) MATLAB
11 1 (0.47%) Shell Script
Qualifications
1 3 (1.42%) AWS Certification
1 3 (1.42%) Degree
1 3 (1.42%) SC Cleared
1 3 (1.42%) Security Cleared
Quality Assurance & Compliance
1 15 (7.08%) Data Quality
2 6 (2.83%) QA
3 5 (2.36%) GDPR
System Software
1 39 (18.40%) Docker
2 3 (1.42%) Active Directory
3 2 (0.94%) Hyper-V
3 2 (0.94%) NFS
3 2 (0.94%) VMware Infrastructure
4 1 (0.47%) Apache Flume
Systems Management
1 42 (19.81%) Kubernetes
2 16 (7.55%) Terraform
3 8 (3.77%) Ansible
4 5 (2.36%) Grafana
5 3 (1.42%) Prometheus
6 2 (0.94%) Active Directory Federation Services
6 2 (0.94%) Opscode Chef
6 2 (0.94%) Puppet
7 1 (0.47%) Oozie
Vendors
1 55 (25.94%) Databricks
2 30 (14.15%) Microsoft
3 14 (6.60%) Google
4 9 (4.25%) Oracle
5 8 (3.77%) Splunk
6 6 (2.83%) Alteryx
7 5 (2.36%) Foundry
7 5 (2.36%) Informatica
7 5 (2.36%) Palantir
8 3 (1.42%) Cloudera
8 3 (1.42%) Dataiku
8 3 (1.42%) Zerto
9 2 (0.94%) Hortonworks
9 2 (0.94%) NetApp
9 2 (0.94%) Red Hat
9 2 (0.94%) StreamSets
9 2 (0.94%) Talend
9 2 (0.94%) VMware
10 1 (0.47%) Teradata
10 1 (0.47%) TOWER Software