Period
to 21 September 2021

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 21 September 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
England > London
6 months to
21 Sep 2021
Same period 2020 Same period 2019
Rank 347 260 395
Rank change year-on-year -87 +135 +155
Contract jobs citing PySpark 208 105 188
As % of all contract jobs advertised in London 0.62% 0.79% 0.49%
As % of the Libraries, Frameworks & Software Standards category 1.78% 2.06% 1.35%
Number of daily rates quoted 138 58 131
10th Percentile £437 £380 £460
25th Percentile £500 £413 £513
Median daily rate (50th Percentile) £566 £523 £550
Median % change year-on-year +8.37% -5.00% -
75th Percentile £650 £584 £588
90th Percentile £695 £663 £638
England median daily rate £550 £520 £550
% change year-on-year +5.77% -5.45% -

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
London
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 11,677 5,089 13,926
As % of all contract IT jobs advertised in London 34.66% 38.42% 36.18%
Number of daily rates quoted 8,746 3,751 10,473
10th Percentile £400 £363 £375
25th Percentile £465 £425 £450
Median daily rate (50th Percentile) £550 £525 £525
Median % change year-on-year +4.76% - +1.45%
75th Percentile £656 £631 £613
90th Percentile £725 £713 £688
England median daily rate £525 £500 £500
% change year-on-year +5.00% - +5.26%
Number of hourly rates quoted 45 25 57
10th Percentile £22.85 £21.90 £15.35
25th Percentile £35.00 £25.00 £33.25
Median hourly rate £49.00 £37.50 £50.00
Median % change year-on-year +30.67% -25.00% +14.61%
75th Percentile £65.00 £48.00 £55.00
90th Percentile £71.00 £55.60 £70.45
England median hourly rate £49.00 £44.50 £46.40
% change year-on-year +10.11% -4.09% +10.68%

PySpark
Job Vacancy Trend in London

Job postings citing PySpark as a proportion of all IT jobs advertised in London.

Job vacancy trend for PySpark in London

PySpark
Contractor Daily Rate Trend in London

3-month moving average daily rate quoted in jobs citing PySpark in London.

Daily rate trend for PySpark in London

PySpark
Daily Rate Histogram in London

Daily rate distribution for jobs citing PySpark in London over the 6 months to 21 September 2021.

Daily rate histogram for PySpark in London

PySpark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the London region over the 6 months to 21 September 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -45 17 £625 +42.86% 11
South London 0 4 £450 -30.77%
West London - 1 £570 -
PySpark
England

For the 6 months to 21 September 2021, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for PySpark.

1 159 (76.44%) Python
2 110 (52.88%) SQL
3 98 (47.12%) Apache Spark
4 86 (41.35%) Azure
5 77 (37.02%) Databricks
6 74 (35.58%) Data Modelling
7 71 (34.13%) Big Data
8 66 (31.73%) Agile Software Development
9 64 (30.77%) Scala
10 59 (28.37%) Git (software)
11 56 (26.92%) Finance
12 53 (25.48%) Analytics
13 51 (24.52%) ETL
14 49 (23.56%) Hadoop
14 49 (23.56%) AWS
15 42 (20.19%) Apache Hive
16 41 (19.71%) Scrum
16 41 (19.71%) Social Skills
17 40 (19.23%) Azure Data Factory
18 38 (18.27%) Business Intelligence
19 37 (17.79%) DevOps
19 37 (17.79%) Data Lake
20 33 (15.87%) Data Engineering
21 32 (15.38%) Data Science
21 32 (15.38%) Banking
22 30 (14.42%) Jenkins
23 29 (13.94%) Power BI
23 29 (13.94%) Power Platform
24 28 (13.46%) Data Pipeline
25 26 (12.50%) Data Ingestion

PySpark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 98 (47.12%) Apache Spark
2 19 (9.13%) Apache
2 19 (9.13%) Apache Airflow
2 19 (9.13%) Confluence
3 10 (4.81%) Elasticsearch
4 3 (1.44%) SAS
5 2 (0.96%) Luigi
6 1 (0.48%) Apache Pig
Cloud Services
1 86 (41.35%) Azure
2 49 (23.56%) AWS
3 40 (19.23%) Azure Data Factory
4 29 (13.94%) Power Platform
5 26 (12.50%) Amazon S3
6 24 (11.54%) Azure Synapse Analytics
7 20 (9.62%) Serverless
8 14 (6.73%) AWS Lambda
9 11 (5.29%) Azure Cosmos DB
10 9 (4.33%) GCP
11 7 (3.37%) Azure Event Grid
12 6 (2.88%) Amazon Kinesis
12 6 (2.88%) Azure Logic Apps
13 5 (2.40%) Azure Functions
14 4 (1.92%) Azure Data Lake Analytics
14 4 (1.92%) Azure Stack
14 4 (1.92%) Collibra
14 4 (1.92%) IaaS
14 4 (1.92%) PaaS
15 3 (1.44%) GitHub
Communications & Networking
1 1 (0.48%) Cisco ASA
1 1 (0.48%) Firewall
Database & Business Intelligence
1 71 (34.13%) Big Data
2 49 (23.56%) Hadoop
3 42 (20.19%) Apache Hive
4 37 (17.79%) Data Lake
5 29 (13.94%) Power BI
6 18 (8.65%) Data Warehouse
7 13 (6.25%) Amazon Athena
7 13 (6.25%) NoSQL
7 13 (6.25%) SQL Server
7 13 (6.25%) Tableau
8 12 (5.77%) Azure SQL Data Warehouse
8 12 (5.77%) Azure SQL Database
9 11 (5.29%) Delta Lake
9 11 (5.29%) Relational Database
10 10 (4.81%) Amazon Redshift
10 10 (4.81%) PostgreSQL
11 9 (4.33%) Metadata
12 8 (3.85%) BigQuery
13 7 (3.37%) MongoDB
14 6 (2.88%) Neo4j
Development Applications
1 59 (28.37%) Git (software)
2 30 (14.42%) Jenkins
3 26 (12.50%) JIRA
4 19 (9.13%) GitLab
5 13 (6.25%) Hudson
6 11 (5.29%) TeamCity
7 5 (2.40%) Bitbucket
8 4 (1.92%) git-flow
9 3 (1.44%) MLflow
10 2 (0.96%) Artifactory
10 2 (0.96%) SonarQube
10 2 (0.96%) Sonatype Nexus
11 1 (0.48%) kappa
11 1 (0.48%) Maven
11 1 (0.48%) Octopus Deploy
11 1 (0.48%) Visual Studio
General
1 56 (26.92%) Finance
2 41 (19.71%) Social Skills
3 32 (15.38%) Banking
4 16 (7.69%) Analytical Skills
5 12 (5.77%) Investment Banking
6 10 (4.81%) Presentation Skills
7 5 (2.40%) Public Sector
8 3 (1.44%) Games
8 3 (1.44%) Retail
9 2 (0.96%) Law
9 2 (0.96%) Pharmaceutical
9 2 (0.96%) Telecoms
10 1 (0.48%) Documentation Skills
10 1 (0.48%) German Language
10 1 (0.48%) Influencing Skills
10 1 (0.48%) Marketing
Job Titles
1 133 (63.94%) Data Engineer
2 43 (20.67%) Developer
3 31 (14.90%) Azure Engineer
4 24 (11.54%) Senior Data Engineer
5 18 (8.65%) Python Engineer
6 14 (6.73%) AWS Engineer
7 10 (4.81%) Architect
7 10 (4.81%) Big Data Engineer
7 10 (4.81%) Data Architect
7 10 (4.81%) Data Scientist
8 9 (4.33%) Python Developer
9 6 (2.88%) Azure Architect
9 6 (2.88%) Big Data Developer
10 5 (2.40%) Customer Engineer
10 5 (2.40%) Data Science Engineer
11 4 (1.92%) Software Engineer
11 4 (1.92%) Test Engineer
12 3 (1.44%) Solutions Architect
13 2 (0.96%) Applications Developer
13 2 (0.96%) Senior Software Engineer
Libraries, Frameworks & Software Standards
1 25 (12.02%) Kafka
2 19 (9.13%) pytest
3 10 (4.81%) Pandas
4 8 (3.85%) Velocity
5 6 (2.88%) ARM Templates
5 6 (2.88%) Spark Streaming
6 5 (2.40%) Django
6 5 (2.40%) JSON
6 5 (2.40%) scikit-learn
6 5 (2.40%) Web Services
7 4 (1.92%) NumPy
7 4 (1.92%) REST
7 4 (1.92%) RESTful
7 4 (1.92%) Shiny
7 4 (1.92%) TensorFlow
8 3 (1.44%) Apache Avro
8 3 (1.44%) Apollo GraphQL
8 3 (1.44%) GraphQL
8 3 (1.44%) MLlib
8 3 (1.44%) PyTorch
Miscellaneous
1 12 (5.77%) Data Structures
2 9 (4.33%) Cloud Native
3 8 (3.85%) Greenfield Project
4 7 (3.37%) Linux Command Line
4 7 (3.37%) Virtual Team
5 3 (1.44%) Algorithms
5 3 (1.44%) iPad
5 3 (1.44%) Renewable Energy
6 2 (0.96%) Credit Risk
6 2 (0.96%) Management Information System
6 2 (0.96%) NHS
6 2 (0.96%) Replication
7 1 (0.48%) Reference Data
7 1 (0.48%) Self-Motivation
7 1 (0.48%) User Experience
Operating Systems
1 16 (7.69%) Unix
2 6 (2.88%) Linux
Processes & Methodologies
1 74 (35.58%) Data Modelling
2 66 (31.73%) Agile Software Development
3 53 (25.48%) Analytics
4 51 (24.52%) ETL
5 41 (19.71%) Scrum
6 38 (18.27%) Business Intelligence
7 37 (17.79%) DevOps
8 33 (15.87%) Data Engineering
9 32 (15.38%) Data Science
10 28 (13.46%) Data Pipeline
11 26 (12.50%) Data Ingestion
12 22 (10.58%) Data Transformation
12 22 (10.58%) SDLC
13 20 (9.62%) Visualisation
14 17 (8.17%) OO
15 16 (7.69%) Software Engineering
16 15 (7.21%) Machine Learning
17 14 (6.73%) Kimball Methodology
18 13 (6.25%) Data Analytics
18 13 (6.25%) TDD
Programming Languages
1 159 (76.44%) Python
2 110 (52.88%) SQL
3 64 (30.77%) Scala
4 12 (5.77%) Java
5 11 (5.29%) R
6 5 (2.40%) JavaScript
7 4 (1.92%) C#
8 3 (1.44%) TypeScript
9 2 (0.96%) Bash Shell
9 2 (0.96%) U-SQL
10 1 (0.48%) C
10 1 (0.48%) C++
10 1 (0.48%) Go
10 1 (0.48%) PHP
10 1 (0.48%) PowerShell
10 1 (0.48%) SPARQL
Qualifications
1 7 (3.37%) Security Cleared
2 2 (0.96%) Computer Science Degree
2 2 (0.96%) Degree
2 2 (0.96%) SC Cleared
3 1 (0.48%) BPSS Clearance
3 1 (0.48%) ISEB
3 1 (0.48%) ISTQB
3 1 (0.48%) PhD
Quality Assurance & Compliance
1 15 (7.21%) Data Quality
2 14 (6.73%) QA
3 2 (0.96%) SLA
4 1 (0.48%) SNOMED CT
System Software
1 19 (9.13%) Docker
2 4 (1.92%) HDFS
3 1 (0.48%) Active Directory
Systems Management
1 15 (7.21%) Ansible
2 12 (5.77%) Kubernetes
3 7 (3.37%) Control-M
4 6 (2.88%) Terraform
5 2 (0.96%) Kubeflow
5 2 (0.96%) Oozie
5 2 (0.96%) Tivoli
6 1 (0.48%) Active Directory Federation Services
6 1 (0.48%) FortiGate
6 1 (0.48%) Hadoop YARN
6 1 (0.48%) Opscode Chef
6 1 (0.48%) Puppet
6 1 (0.48%) Salt
Vendors
1 77 (37.02%) Databricks
2 24 (11.54%) Microsoft
3 21 (10.10%) Cloudera
4 20 (9.62%) Hortonworks
5 10 (4.81%) Oracle
5 10 (4.81%) Teradata
6 9 (4.33%) Informatica
7 7 (3.37%) Netezza
8 6 (2.88%) Confluent
9 4 (1.92%) Google
10 2 (0.96%) ServiceNow
10 2 (0.96%) StreamSets
11 1 (0.48%) CheckPoint
11 1 (0.48%) Cisco