Period
to 25 September 2020

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 25 September 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
25 Sep 2020
Same period 2019 Same period 2018
Rank 190 164 195
Rank change year-on-year -26 +31 +3
Contract jobs citing Apache Spark 420 1,458 1,512
As % of all contract jobs advertised in the UK 1.41% 1.65% 1.47%
As % of the Application Platforms category 9.45% 10.89% 8.78%
Number of daily rates quoted 276 1,079 1,106
Median daily rate £525 £550 £550
Median daily rate % change year-on-year -4.55% - -
10th Percentile £363 £425 £425
90th Percentile £688 £713 £700
UK excluding London median daily rate £475 £525 £500
% change year-on-year -9.52% +5.00% +8.23%
Number of hourly rates quoted 17 3 8
Median hourly rate £35.00 £24.65 £33.75
Median hourly rate % change year-on-year +41.99% -26.96% +74.51%
UK excluding London median hourly rate £35.00 £24.65 £22.00
% change year-on-year +41.99% +12.05% +13.75%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 4,446 13,389 17,212
As % of all contract IT jobs advertised in the UK 14.96% 15.16% 16.74%
Number of daily rates quoted 2,773 8,530 11,265
Median daily rate £470 £460 £450
Median daily rate % change year-on-year +2.17% +2.22% +2.86%
10th Percentile £294 £263 £270
90th Percentile £656 £638 £625
UK excluding London median daily rate £440 £425 £425
% change year-on-year +3.53% - +6.25%
Number of hourly rates quoted 148 326 423
Median hourly rate £25.65 £21.50 £21.00
Median hourly rate % change year-on-year +19.30% +2.38% +5.00%
10th Percentile £12.63 £11.25 £12.03
90th Percentile £54.49 £52.38 £51.00
Median hourly rate £22.91 £21.00 £20.50
% change year-on-year +9.10% +2.44% +2.50%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 25 September 2020.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 25 September 2020.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 25 September 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -20 378 £538 -2.18% 109
London -24 280 £550 -4.35% 83
UK excluding London +27 105 £475 -9.52% 37
Work from Home -58 47 £539 -20.15% 25
South East +121 41 £475 -5.00% 14
East of England +67 21 £631 +20.24% 2
North of England +46 19 £413 -23.43% 6
North West +47 17 £400 -24.88% 2
Wales +48 11 £500 -18.37% 2
Scotland +1 9 £410 -16.33% 9
South West +93 4 £463 -19.57% 2
Midlands +76 4 £400 -22.03% 2
East Midlands +55 3 - -
Yorkshire +40 2 £425 -20.00% 4
West Midlands +54 1 £400 -20.00% 2
Northern Ireland +12 1 £350 -

For the 6 months to 25 September 2020, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 259 (61.67%) Python
2 258 (61.43%) Big Data
3 233 (55.48%) Hadoop
4 179 (42.62%) SQL
5 166 (39.52%) Scala
6 147 (35.00%) Java
7 144 (34.29%) AWS
8 138 (32.86%) Agile Software Development
9 133 (31.67%) Kafka
10 128 (30.48%) Apache Hive
11 126 (30.00%) Azure
12 109 (25.95%) Analytics
13 105 (25.00%) ETL
14 87 (20.71%) DevOps
15 83 (19.76%) Finance
16 82 (19.52%) Data Science
17 78 (18.57%) NoSQL
18 77 (18.33%) GCP
19 73 (17.38%) Kubernetes
19 73 (17.38%) Machine Learning
20 72 (17.14%) PySpark
21 67 (15.95%) Microservices
22 66 (15.71%) Impala
23 65 (15.48%) Git (software)
24 64 (15.24%) Cloudera
24 64 (15.24%) Docker
24 64 (15.24%) Security Cleared
25 63 (15.00%) Apache
26 59 (14.05%) Data Modelling
27 58 (13.81%) Computer Science

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 63 (15.00%) Apache
2 25 (5.95%) Elasticsearch
3 20 (4.76%) SAS
4 19 (4.52%) Apache Pig
5 15 (3.57%) Apache Airflow
6 9 (2.14%) Confluence
7 6 (1.43%) Jupyter
8 5 (1.19%) Tornado
9 2 (0.48%) Luigi
10 1 (0.24%) Apache Solr
10 1 (0.24%) Camunda
10 1 (0.24%) nginx
10 1 (0.24%) Oracle Workflow
10 1 (0.24%) R Markdown
10 1 (0.24%) Tomcat
Applications
1 1 (0.24%) Microsoft Excel
1 1 (0.24%) Microsoft PowerPoint
Business Applications
1 4 (0.95%) SAP Oil and Gas
2 1 (0.24%) FINCAD Analytics
2 1 (0.24%) Oracle BPM
2 1 (0.24%) Sparx Enterprise Architect
Cloud Services
1 144 (34.29%) AWS
2 126 (30.00%) Azure
3 77 (18.33%) GCP
4 26 (6.19%) Amazon EMR
5 25 (5.95%) Amazon S3
6 24 (5.71%) Azure Data Factory
7 23 (5.48%) Serverless
8 16 (3.81%) Amazon EC2
9 15 (3.57%) Amazon SQS
10 13 (3.10%) Cloud Computing
11 12 (2.86%) AWS Lambda
11 12 (2.86%) Power Platform
12 10 (2.38%) Virtual Private Cloud
13 9 (2.14%) Amazon Kinesis
14 8 (1.90%) Amazon EKS
14 8 (1.90%) SaaS
15 6 (1.43%) Azure Functions
16 5 (1.19%) Azure Active Directory
16 5 (1.19%) PaaS
17 4 (0.95%) AWS CloudFormation
Communications & Networking
1 30 (7.14%) SSL
2 25 (5.95%) HTTPS
3 7 (1.67%) HTTP
3 7 (1.67%) Kerberos
4 3 (0.71%) VPN
5 2 (0.48%) RTMP
5 2 (0.48%) Skype
6 1 (0.24%) Internet
6 1 (0.24%) Wi-Fi
6 1 (0.24%) Wireless
Database & Business Intelligence
1 258 (61.43%) Big Data
2 233 (55.48%) Hadoop
3 128 (30.48%) Apache Hive
4 78 (18.57%) NoSQL
5 66 (15.71%) Impala
6 39 (9.29%) Relational Database
7 36 (8.57%) Data Lake
8 32 (7.62%) Data Warehouse
9 30 (7.14%) HBase
10 26 (6.19%) Amazon Redshift
10 26 (6.19%) MongoDB
11 25 (5.95%) PostgreSQL
12 23 (5.48%) Apache Cassandra
13 21 (5.00%) Spark SQL
14 19 (4.52%) Amazon RDS
15 15 (3.57%) DataStage
15 15 (3.57%) Sqoop
16 14 (3.33%) BigQuery
16 14 (3.33%) Data Mining
17 13 (3.10%) MySQL
Development Applications
1 65 (15.48%) Git (software)
2 39 (9.29%) Jenkins
3 15 (3.57%) GitLab
4 14 (3.33%) JIRA
5 10 (2.38%) TeamCity
6 9 (2.14%) Artifactory
7 7 (1.67%) Gradle
7 7 (1.67%) Maven
8 6 (1.43%) ProC
9 5 (1.19%) Hudson
9 5 (1.19%) kappa
10 3 (0.71%) JUnit
10 3 (0.71%) SonarQube
10 3 (0.71%) Visual Studio Team System
11 2 (0.48%) Gatling
11 2 (0.48%) Locust
11 2 (0.48%) Mockito
11 2 (0.48%) NUnit
11 2 (0.48%) Sonatype Nexus
11 2 (0.48%) Travis CI
General
1 83 (19.76%) Finance
2 56 (13.33%) Banking
3 26 (6.19%) Front Office
4 17 (4.05%) Investment Banking
5 15 (3.57%) Public Sector
6 14 (3.33%) Marketing
6 14 (3.33%) Retail
7 11 (2.62%) Pharmaceutical
8 10 (2.38%) Legal
9 9 (2.14%) Manufacturing
9 9 (2.14%) Telecoms
10 7 (1.67%) Electronics
11 5 (1.19%) Aerospace
11 5 (1.19%) Financial Institution
12 4 (0.95%) Games
13 3 (0.71%) Law
14 2 (0.48%) Automotive
14 2 (0.48%) Military
15 1 (0.24%) Billing
15 1 (0.24%) Retail Banking
Job Titles
1 167 (39.76%) Data Engineer
2 127 (30.24%) Developer
3 57 (13.57%) Big Data Engineer
4 48 (11.43%) Java Developer
5 45 (10.71%) Python Engineer
6 33 (7.86%) Azure Engineer
7 31 (7.38%) Data Scientist
8 26 (6.19%) Architect
9 25 (5.95%) Scala Developer
10 21 (5.00%) Python Developer
11 20 (4.76%) Senior Data Engineer
11 20 (4.76%) Senior Developer
12 18 (4.29%) Big Data Developer
13 17 (4.05%) Hadoop Engineer
14 15 (3.57%) AWS Engineer
15 12 (2.86%) Data Architect
15 12 (2.86%) Java Engineer
15 12 (2.86%) Technical Leader
16 11 (2.62%) Apache Spark Developer
16 11 (2.62%) Consultant
Libraries, Frameworks & Software Standards
1 133 (31.67%) Kafka
2 72 (17.14%) PySpark
3 44 (10.48%) Spring
4 28 (6.67%) RESTful
5 27 (6.43%) Spring Security
6 25 (5.95%) TensorFlow
7 24 (5.71%) JAAS
8 22 (5.24%) Spark Streaming
9 21 (5.00%) Akka
10 20 (4.76%) Guice
10 20 (4.76%) scikit-learn
11 16 (3.81%) Elastic Stack
12 15 (3.57%) Pandas
13 12 (2.86%) .NET
13 12 (2.86%) Keras
13 12 (2.86%) Spring Boot
14 11 (2.62%) HTML
15 10 (2.38%) NumPy
16 9 (2.14%) CSS
16 9 (2.14%) JSON
Miscellaneous
1 57 (13.57%) Analytical Skills
2 33 (7.86%) Algorithms
3 22 (5.24%) Greenfield Project
4 14 (3.33%) Foreign Exchange (FX)
5 12 (2.86%) FMCG
6 11 (2.62%) Credit Risk
7 10 (2.38%) Cloud Native
7 10 (2.38%) Data Structures
7 10 (2.38%) Virtual Team
8 9 (2.14%) Analytical Mindset
8 9 (2.14%) Enterprise Software
9 8 (1.90%) PMI
9 8 (1.90%) Reference Data
10 6 (1.43%) Self-Motivation
11 5 (1.19%) Life Science
12 4 (0.95%) Data Centre
12 4 (0.95%) Fixed Income
12 4 (0.95%) Grid Computing
12 4 (0.95%) Management Information System
12 4 (0.95%) Market Risk
Operating Systems
1 34 (8.10%) Linux
2 11 (2.62%) Unix
3 6 (1.43%) Debian
3 6 (1.43%) Red Hat Enterprise Linux
4 4 (0.95%) Ubuntu
4 4 (0.95%) Windows
5 2 (0.48%) CentOS
Processes & Methodologies
1 138 (32.86%) Agile Software Development
2 109 (25.95%) Analytics
3 105 (25.00%) ETL
4 87 (20.71%) DevOps
5 82 (19.52%) Data Science
6 73 (17.38%) Machine Learning
7 67 (15.95%) Microservices
8 59 (14.05%) Data Modelling
9 58 (13.81%) Computer Science
10 55 (13.10%) Continuous Integration
11 54 (12.86%) Visualisation
12 53 (12.62%) Data Engineering
13 51 (12.14%) Mathematics
14 48 (11.43%) Business Intelligence
15 42 (10.00%) Data Visualisation
16 41 (9.76%) TDD
16 41 (9.76%) Test Automation
17 38 (9.05%) SDLC
18 37 (8.81%) Scrum
19 35 (8.33%) Software Engineering
Programming Languages
1 259 (61.67%) Python
2 179 (42.62%) SQL
3 166 (39.52%) Scala
4 147 (35.00%) Java
5 36 (8.57%) R
6 30 (7.14%) Java 8
7 29 (6.90%) JavaScript
8 26 (6.19%) Kotlin
9 24 (5.71%) C#
10 18 (4.29%) Shell Script
11 12 (2.86%) Haskell
12 11 (2.62%) Ruby
13 10 (2.38%) Perl
14 7 (1.67%) Bash Shell
14 7 (1.67%) C++
14 7 (1.67%) Go
14 7 (1.67%) Groovy
15 6 (1.43%) PHP
16 4 (0.95%) Data Analysis Expressions (DAX)
16 4 (0.95%) T-SQL
Qualifications
1 64 (15.24%) Security Cleared
2 53 (12.62%) SC Cleared
3 52 (12.38%) Degree
4 22 (5.24%) Numerate Degree
5 17 (4.05%) Computer Science Degree
6 11 (2.62%) Master's Degree
7 8 (1.90%) PMI Certification
8 6 (1.43%) DV Cleared
9 4 (0.95%) Microsoft Certification
9 4 (0.95%) PhD
10 2 (0.48%) AWS Certification
11 1 (0.24%) CSPO
11 1 (0.24%) Postgraduate
Quality Assurance & Compliance
1 7 (1.67%) MiFID
2 5 (1.19%) Data Quality
2 5 (1.19%) GxP
3 3 (0.71%) QA
3 3 (0.71%) SLA
4 2 (0.48%) Sarbanes-Oxley
5 1 (0.24%) CDISC
System Software
1 64 (15.24%) Docker
2 50 (11.90%) HDFS
3 11 (2.62%) Apache ZooKeeper
4 10 (2.38%) Apache Flume
5 5 (1.19%) Active Directory
6 2 (0.48%) Active Directory Lightweight Directory Services
Systems Management
1 73 (17.38%) Kubernetes
2 25 (5.95%) Terraform
3 22 (5.24%) Ansible
3 22 (5.24%) Prometheus
3 22 (5.24%) Zipkin
4 14 (3.33%) Oozie
5 13 (3.10%) Hadoop YARN
6 8 (1.90%) Puppet
7 6 (1.43%) Opscode Chef
8 2 (0.48%) Ambari
8 2 (0.48%) Fluentd
8 2 (0.48%) Knative
8 2 (0.48%) logstash
9 1 (0.24%) Grafana
9 1 (0.24%) Istio
9 1 (0.24%) Single Sign-On
9 1 (0.24%) vRealize Orchestrator
Vendors
1 64 (15.24%) Cloudera
2 51 (12.14%) Databricks
3 36 (8.57%) Tableau
4 22 (5.24%) Hortonworks
5 21 (5.00%) Google
5 21 (5.00%) Teradata
6 20 (4.76%) Oracle
7 17 (4.05%) Talend
8 16 (3.81%) Microsoft
9 15 (3.57%) StreamSets
10 10 (2.38%) SAP
11 8 (1.90%) Red Hat
12 7 (1.67%) MapR
12 7 (1.67%) MuleSoft
12 7 (1.67%) Splunk
13 6 (1.43%) IBM
13 6 (1.43%) Informatica
14 5 (1.19%) Salesforce.com
15 4 (0.95%) Ab Initio
15 4 (0.95%) Alteryx