Period
to 19 October 2021

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 19 October 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
Central London > City of London
6 months to
19 Oct 2021
Same period 2020 Same period 2019
Rank 156 104 179
Rank change year-on-year -52 +75 +30
Contract jobs citing PySpark 18 10 49
As % of all contract jobs advertised in the City of London 0.46% 0.56% 0.83%
As % of the Libraries, Frameworks & Software Standards category 1.38% 1.88% 2.31%
Number of daily rates quoted 14 9 47
10th Percentile £469 £405 "
25th Percentile £537 £408 £513
Median daily rate (50th Percentile) £625 £438 £525
Median % change year-on-year +42.86% -16.67% -4.55%
75th Percentile £689 £530 £588
90th Percentile £830 £575 £700
Central London median daily rate £625 £438 £555
% change year-on-year +42.86% -21.17% +0.91%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
City of London
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 1,300 533 2,121
As % of all contract IT jobs advertised in the City of London 33.44% 29.86% 35.74%
Number of daily rates quoted 999 425 1,733
10th Percentile £413 £363 £406
25th Percentile £483 £425 £463
Median daily rate (50th Percentile) £600 £525 £533
Median % change year-on-year +14.29% -1.50% +1.52%
75th Percentile £675 £613 £618
90th Percentile £750 £688 £688
Central London median daily rate £590 £525 £525
% change year-on-year +12.38% - -
Number of hourly rates quoted 8 4 6
10th Percentile £18.78 £26.28 £31.06
25th Percentile £24.25 £30.44 £41.23
Median hourly rate £35.75 £35.50 £46.85
Median % change year-on-year +0.70% -24.22% +112.94%
75th Percentile £45.25 £38.69 £53.91
90th Percentile £68.43 £39.48 £60.25
Central London median hourly rate £35.00 £35.50 £46.85
% change year-on-year -1.41% -24.22% +112.94%

PySpark
Job Vacancy Trend in the City of London

Job postings citing PySpark as a proportion of all IT jobs advertised in the City of London.

Job vacancy trend for PySpark in the City of London

PySpark
Contractor Daily Rate Trend in the City of London

3-month moving average daily rate quoted in jobs citing PySpark in the City of London.

Daily rate trend for PySpark in the City of London

PySpark
Daily Rate Histogram in the City of London

Daily rate distribution for jobs citing PySpark in the City of London over the 6 months to 19 October 2021.

Daily rate histogram for PySpark in the City of London

For the 6 months to 19 October 2021, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for PySpark.

1 12 (66.67%) Apache Spark
2 11 (61.11%) Apache Hive
3 10 (55.56%) Scala
3 10 (55.56%) Banking
3 10 (55.56%) Finance
3 10 (55.56%) Big Data
4 9 (50.00%) Python
5 7 (38.89%) SQL
5 7 (38.89%) Hortonworks
5 7 (38.89%) Cloudera
6 6 (33.33%) ETL
7 5 (27.78%) Hadoop
8 4 (22.22%) Data Modelling
8 4 (22.22%) Databricks
9 3 (16.67%) Azure
9 3 (16.67%) Analytics
9 3 (16.67%) Elasticsearch
9 3 (16.67%) Agile Software Development
9 3 (16.67%) Azure Data Factory
9 3 (16.67%) Infrastructure as Code
9 3 (16.67%) Delta Lake
9 3 (16.67%) Problem-Solving
10 2 (11.11%) Technical Leadership
10 2 (11.11%) Azure Synapse Analytics
10 2 (11.11%) Power Platform
10 2 (11.11%) Architectural Design
10 2 (11.11%) Confluent
10 2 (11.11%) Serverless
10 2 (11.11%) Data Science
10 2 (11.11%) Microsoft

PySpark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 12 (66.67%) Apache Spark
2 3 (16.67%) Elasticsearch
3 2 (11.11%) Apache
3 2 (11.11%) Apache Airflow
Cloud Services
1 3 (16.67%) Azure
1 3 (16.67%) Azure Data Factory
2 2 (11.11%) Amazon S3
2 2 (11.11%) AWS
2 2 (11.11%) AWS Lambda
2 2 (11.11%) Azure Synapse Analytics
2 2 (11.11%) Power Platform
2 2 (11.11%) Serverless
3 1 (5.56%) Amazon EC2
3 1 (5.56%) Amazon EMR
3 1 (5.56%) Collibra
Database & Business Intelligence
1 11 (61.11%) Apache Hive
2 10 (55.56%) Big Data
3 5 (27.78%) Hadoop
4 3 (16.67%) Delta Lake
5 2 (11.11%) Azure SQL Database
5 2 (11.11%) Data Lake
5 2 (11.11%) Data Warehouse
5 2 (11.11%) Microsoft Analysis Services
5 2 (11.11%) NoSQL
5 2 (11.11%) Power BI
5 2 (11.11%) SQL Server
5 2 (11.11%) SQL Server Analysis Services
6 1 (5.56%) Amazon Athena
6 1 (5.56%) Amazon DynamoDB
6 1 (5.56%) Amazon RDS
6 1 (5.56%) Dremio
6 1 (5.56%) Metadata
Development Applications
1 1 (5.56%) Git (software)
1 1 (5.56%) git-flow
1 1 (5.56%) Jenkins
General
1 10 (55.56%) Banking
1 10 (55.56%) Finance
2 1 (5.56%) Documentation Skills
2 1 (5.56%) Social Skills
Job Titles
1 11 (61.11%) Data Engineer
2 3 (16.67%) Developer
3 1 (5.56%) AWS Engineer
3 1 (5.56%) Data Modeller
3 1 (5.56%) Data Scientist
3 1 (5.56%) Kubernetes Engineer
3 1 (5.56%) Python Engineer
Libraries, Frameworks & Software Standards
1 2 (11.11%) Apache Avro
1 2 (11.11%) JDBC
1 2 (11.11%) JMS
1 2 (11.11%) JSON
1 2 (11.11%) Kafka
2 1 (5.56%) Elastic Stack
2 1 (5.56%) PyArrow
Miscellaneous
1 2 (11.11%) Management Information System
2 1 (5.56%) Credit Risk
Operating Systems
1 1 (5.56%) Linux
1 1 (5.56%) Unix
Processes & Methodologies
1 6 (33.33%) ETL
2 4 (22.22%) Data Modelling
3 3 (16.67%) Agile Software Development
3 3 (16.67%) Analytics
3 3 (16.67%) Infrastructure as Code
3 3 (16.67%) Problem-Solving
4 2 (11.11%) Architectural Design
4 2 (11.11%) Business Intelligence
4 2 (11.11%) Data Analytics
4 2 (11.11%) Data Engineering
4 2 (11.11%) Data Integration
4 2 (11.11%) Data Management
4 2 (11.11%) Data Science
4 2 (11.11%) Data Strategy
4 2 (11.11%) Data Transformation
4 2 (11.11%) Kanban
4 2 (11.11%) Kimball Methodology
4 2 (11.11%) Machine Learning
4 2 (11.11%) Scrum
4 2 (11.11%) Technical Leadership
Programming Languages
1 10 (55.56%) Scala
2 9 (50.00%) Python
3 7 (38.89%) SQL
4 1 (5.56%) R
Quality Assurance & Compliance
1 1 (5.56%) IFRS
1 1 (5.56%) IFRS 9
1 1 (5.56%) SLA
System Software
1 1 (5.56%) HDFS
Systems Management
1 1 (5.56%) Ansible
1 1 (5.56%) Kibana
1 1 (5.56%) Kubernetes
1 1 (5.56%) Terraform
Vendors
1 7 (38.89%) Cloudera
1 7 (38.89%) Hortonworks
2 4 (22.22%) Databricks
3 2 (11.11%) Confluent
3 2 (11.11%) Microsoft