Period
to 8 April 2020

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 8 April 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
8 Apr 2020
Same period 2019 Same period 2018
Rank 216 176 181
Rank change year-on-year -40 +5 +31
Contract jobs citing Apache Spark 722 1,563 1,543
As % of all contract jobs advertised in the UK 1.18% 1.61% 1.52%
As % of the Application Platforms category 8.21% 10.03% 9.25%
Number of daily rates quoted 519 1,114 1,113
Median daily rate £550 £550 £550
10th Percentile £413 £425 £413
90th Percentile £700 £713 £675
UK excluding London median daily rate £525 £525 £500
% change year-on-year - +5.00% -
Number of hourly rates quoted 1 2 7
Median hourly rate £52.50 £115.75 £30.50
Median hourly rate % change year-on-year -54.64% +279.51% -69.50%
10th Percentile £51.25 £64.88 £19.34
90th Percentile £53.75 £164.83 £65.75
UK excluding London median hourly rate - - £29.00

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 8,791 15,586 16,689
As % of all contract IT jobs advertised in the UK 14.41% 16.08% 16.40%
Number of daily rates quoted 5,502 10,032 11,063
Median daily rate £460 £475 £450
Median daily rate % change year-on-year -3.16% +5.56% +5.88%
10th Percentile £263 £281 £263
90th Percentile £644 £638 £625
UK excluding London median daily rate £425 £430 £408
% change year-on-year -1.16% +5.52% +4.49%
Number of hourly rates quoted 228 311 374
Median hourly rate £22.50 £18.76 £18.80
Median hourly rate % change year-on-year +19.94% -0.19% -1.08%
10th Percentile £11.89 £12.00 £11.25
90th Percentile £53.00 £48.75 £47.50
Median hourly rate £19.90 £17.00 £17.50
% change year-on-year +17.06% -2.86% -7.89%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 8 April 2020.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 8 April 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -48 649 £550 - 215
London -19 442 £575 - 169
UK excluding London -38 274 £525 - 55
South East -25 83 £500 -4.76% 17
North of England -26 53 £538 +2.38% 4
Scotland +19 37 £575 +15.00% 6
Wales +47 30 £500 -4.76% 3
South West +9 29 £550 - 2
Yorkshire -17 28 £475 -5.00% 1
Midlands +60 22 £525 +7.69% 3
East of England +18 20 £475 -9.00% 20
West Midlands +73 19 £525 +6.06% 2
North West +11 19 £550 -8.33% 3
Work from Home -5 16 £595 +10.70% 14
North East +28 5 £650 +30.00%
East Midlands +15 3 £450 -5.26% 1

For the 6 months to 8 April 2020, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 471 (65.24%) Big Data
2 460 (63.71%) Python
3 398 (55.12%) Hadoop
4 296 (41.00%) SQL
5 284 (39.34%) Scala
6 281 (38.92%) Kafka
7 276 (38.23%) AWS
8 251 (34.76%) Java
9 246 (34.07%) Agile Software Development
10 200 (27.70%) Data Science
11 197 (27.29%) Analytics
12 195 (27.01%) Apache Hive
13 171 (23.68%) Machine Learning
13 171 (23.68%) Azure
14 163 (22.58%) NoSQL
15 141 (19.53%) ETL
16 138 (19.11%) Finance
17 132 (18.28%) Apache
18 131 (18.14%) GCP
19 125 (17.31%) Kubernetes
20 123 (17.04%) Data Modelling
21 122 (16.90%) PySpark
22 121 (16.76%) Git (software)
23 117 (16.20%) HBase
23 117 (16.20%) HDFS
24 110 (15.24%) Docker
25 108 (14.96%) DevOps
26 104 (14.40%) Computer Science
27 101 (13.99%) Linux
28 96 (13.30%) Data Engineering

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 132 (18.28%) Apache
2 39 (5.40%) Apache Airflow
3 34 (4.71%) Elasticsearch
4 29 (4.02%) Jupyter
5 18 (2.49%) SAS
6 14 (1.94%) Apache Pig
7 10 (1.39%) Confluence
7 10 (1.39%) OpenStack
8 9 (1.25%) Luigi
8 9 (1.25%) Tomcat
9 8 (1.11%) nginx
10 6 (0.83%) Cloud Foundry
10 6 (0.83%) DNN
11 5 (0.69%) GemFire
11 5 (0.69%) WebSphere
12 4 (0.55%) Tornado
13 3 (0.42%) Oracle Workflow
14 2 (0.28%) IIS
14 2 (0.28%) Jetty
14 2 (0.28%) WebLogic
Applications
1 4 (0.55%) MS Visio
2 1 (0.14%) Chatbot
2 1 (0.14%) Microsoft Excel
2 1 (0.14%) Microsoft Project
2 1 (0.14%) SPSS
Business Applications
1 5 (0.69%) SAP CRM
1 5 (0.69%) SAP ERP
1 5 (0.69%) SAP IS-U
2 3 (0.42%) FINCAD Analytics
3 2 (0.28%) Distributed Ledger
3 2 (0.28%) NetSuite
3 2 (0.28%) SAP Oil and Gas
4 1 (0.14%) Shopify
4 1 (0.14%) Sparx Enterprise Architect
Cloud Services
1 276 (38.23%) AWS
2 171 (23.68%) Azure
3 131 (18.14%) GCP
4 59 (8.17%) Amazon Kinesis
5 53 (7.34%) Amazon S3
6 46 (6.37%) Serverless
7 40 (5.54%) AWS Lambda
8 38 (5.26%) Amazon EMR
9 30 (4.16%) GitHub
10 25 (3.46%) Power Platform
11 22 (3.05%) Amazon EC2
12 19 (2.63%) Cloud Computing
13 18 (2.49%) Azure Machine Learning
14 17 (2.35%) OpenShift
14 17 (2.35%) PaaS
15 14 (1.94%) Azure Data Factory
16 12 (1.66%) AWS CloudFormation
17 11 (1.52%) Amazon Glacier
17 11 (1.52%) Amazon SageMaker
17 11 (1.52%) CloudFront
Communications & Networking
1 22 (3.05%) Kerberos
2 20 (2.77%) SSL
3 12 (1.66%) DNS
3 12 (1.66%) HTTPS
4 10 (1.39%) HTTP
5 9 (1.25%) RTMP
6 6 (0.83%) Firewall
6 6 (0.83%) VPN
7 5 (0.69%) DSL
7 5 (0.69%) Internet
8 4 (0.55%) TCP/IP
9 3 (0.42%) SAN
10 2 (0.28%) 5G
11 1 (0.14%) MQTT
11 1 (0.14%) Skype
Database & Business Intelligence
1 471 (65.24%) Big Data
2 398 (55.12%) Hadoop
3 195 (27.01%) Apache Hive
4 163 (22.58%) NoSQL
5 117 (16.20%) HBase
6 87 (12.05%) Impala
7 75 (10.39%) MongoDB
8 72 (9.97%) Data Warehouse
9 71 (9.83%) Relational Database
10 70 (9.70%) Apache Cassandra
11 63 (8.73%) Data Lake
12 60 (8.31%) SQL Server
13 52 (7.20%) Amazon Redshift
14 47 (6.51%) Amazon Athena
15 43 (5.96%) Flink
15 43 (5.96%) Spark SQL
16 40 (5.54%) PostgreSQL
17 37 (5.12%) Data Mining
18 33 (4.57%) BigQuery
19 27 (3.74%) Sqoop
Development Applications
1 121 (16.76%) Git (software)
2 86 (11.91%) Jenkins
3 40 (5.54%) Maven
4 38 (5.26%) JIRA
5 26 (3.60%) Sonatype Nexus
6 18 (2.49%) Selenium
7 16 (2.22%) Cucumber
7 16 (2.22%) IBM UrbanCode
8 15 (2.08%) Artifactory
8 15 (2.08%) TeamCity
9 12 (1.66%) JUnit
9 12 (1.66%) Visual Studio Team System
10 11 (1.52%) Gradle
11 10 (1.39%) Mockito
12 9 (1.25%) NUnit
13 8 (1.11%) CircleCI
13 8 (1.11%) Team Foundation Server
14 7 (0.97%) Bitbucket
14 7 (0.97%) ERwin
14 7 (0.97%) GitLab
General
1 138 (19.11%) Finance
2 82 (11.36%) Banking
3 34 (4.71%) Investment Banking
4 33 (4.57%) Retail
5 26 (3.60%) Front Office
6 20 (2.77%) Public Sector
7 19 (2.63%) Telecoms
8 18 (2.49%) Marketing
9 10 (1.39%) Pharmaceutical
10 7 (0.97%) Retail Banking
11 6 (0.83%) Games
12 5 (0.69%) Legal
13 3 (0.42%) Manufacturing
14 2 (0.28%) Aerospace
14 2 (0.28%) Automotive
14 2 (0.28%) Financial Institution
15 1 (0.14%) Corporate Banking
15 1 (0.14%) Dutch Language
15 1 (0.14%) Publishing
Job Titles
1 255 (35.32%) Data Engineer
2 170 (23.55%) Developer
3 106 (14.68%) Data Scientist
4 101 (13.99%) Big Data Engineer
5 82 (11.36%) Architect
6 61 (8.45%) Data Architect
7 51 (7.06%) AWS Engineer
8 49 (6.79%) Java Developer
9 47 (6.51%) Scala Developer
10 45 (6.23%) DevOps Engineer
11 43 (5.96%) Senior Data Scientist
12 38 (5.26%) Python Engineer
13 33 (4.57%) Python Developer
14 29 (4.02%) Big Data Developer
15 25 (3.46%) Big Data Architect
15 25 (3.46%) Hadoop Developer
16 24 (3.32%) Hadoop Engineer
16 24 (3.32%) Platform Engineer
16 24 (3.32%) Senior Developer
17 21 (2.91%) Azure Engineer
Libraries, Frameworks & Software Standards
1 281 (38.92%) Kafka
2 122 (16.90%) PySpark
3 73 (10.11%) TensorFlow
4 67 (9.28%) Pandas
5 54 (7.48%) Spring
6 45 (6.23%) Spark Streaming
7 40 (5.54%) RESTful
8 38 (5.26%) React
9 36 (4.99%) Spring Boot
10 35 (4.85%) AngularJS
10 35 (4.85%) NumPy
10 35 (4.85%) scikit-learn
11 30 (4.16%) MLlib
12 29 (4.02%) Natural Language Processing
13 28 (3.88%) Web Services
14 27 (3.74%) Node.js
15 26 (3.60%) SciPy
16 25 (3.46%) .NET
16 25 (3.46%) Keras
17 20 (2.77%) Apache Beam
Miscellaneous
1 92 (12.74%) Analytical Skills
2 72 (9.97%) Algorithms
3 30 (4.16%) Data Structures
4 25 (3.46%) Greenfield Project
5 20 (2.77%) Self-Motivation
6 19 (2.63%) Foreign Exchange (FX)
7 17 (2.35%) Public Cloud
8 16 (2.22%) Management Information System
9 14 (1.94%) Credit Risk
9 14 (1.94%) User Experience
10 10 (1.39%) Data Centre
10 10 (1.39%) Market Risk
10 10 (1.39%) Social Media
11 8 (1.11%) Hedge funds
11 8 (1.11%) Linux Command Line
12 7 (0.97%) Analytical Mindset
12 7 (0.97%) Fixed Income
12 7 (0.97%) Life Science
13 6 (0.83%) Cloud Native
13 6 (0.83%) Enterprise Software
Operating Systems
1 101 (13.99%) Linux
2 29 (4.02%) Unix
3 20 (2.77%) Windows
4 10 (1.39%) Red Hat Enterprise Linux
5 7 (0.97%) CentOS
6 4 (0.55%) Debian
7 3 (0.42%) Ubuntu
7 3 (0.42%) Windows Server
8 2 (0.28%) Android
8 2 (0.28%) Apple iOS
9 1 (0.14%) DC/OS
Processes & Methodologies
1 246 (34.07%) Agile Software Development
2 200 (27.70%) Data Science
3 197 (27.29%) Analytics
4 171 (23.68%) Machine Learning
5 141 (19.53%) ETL
6 123 (17.04%) Data Modelling
7 108 (14.96%) DevOps
8 104 (14.40%) Computer Science
9 96 (13.30%) Data Engineering
10 87 (12.05%) Business Intelligence
11 83 (11.50%) Software Engineering
12 81 (11.22%) Visualisation
13 78 (10.80%) Artificial Intelligence
14 75 (10.39%) Microservices
15 72 (9.97%) Use Case
16 71 (9.83%) Mathematics
16 71 (9.83%) TDD
17 67 (9.28%) Statistics
18 66 (9.14%) Scrum
19 59 (8.17%) Continuous Integration
Programming Languages
1 460 (63.71%) Python
2 296 (41.00%) SQL
3 284 (39.34%) Scala
4 251 (34.76%) Java
5 66 (9.14%) R
6 42 (5.82%) JavaScript
7 40 (5.54%) C#
8 36 (4.99%) Bash Shell
9 35 (4.85%) Ruby
10 31 (4.29%) Go
11 24 (3.32%) C
12 18 (2.49%) Java 8
12 18 (2.49%) Shell Script
13 17 (2.35%) C++
14 14 (1.94%) Kotlin
14 14 (1.94%) T-SQL
15 12 (1.66%) Perl
16 11 (1.52%) PL/SQL
17 10 (1.39%) PHP
17 10 (1.39%) PowerShell
Qualifications
1 87 (12.05%) Degree
2 29 (4.02%) Computer Science Degree
3 28 (3.88%) PhD
4 27 (3.74%) Security Cleared
5 26 (3.60%) Master's Degree
6 19 (2.63%) SC Cleared
7 14 (1.94%) Numerate Degree
8 11 (1.52%) AWS Certification
9 4 (0.55%) Microsoft Certification
9 4 (0.55%) PMI Certification
10 3 (0.42%) BPSS Clearance
10 3 (0.42%) DBS Check
10 3 (0.42%) Postgraduate
11 2 (0.28%) CompTIA Security+
11 2 (0.28%) GIAC
11 2 (0.28%) Linux+ Certification
11 2 (0.28%) MCSE
11 2 (0.28%) Network+ Certification
11 2 (0.28%) SANS
11 2 (0.28%) SSCP
Quality Assurance & Compliance
1 23 (3.19%) Data Quality
2 15 (2.08%) QA
3 4 (0.55%) PCI DSS
3 4 (0.55%) SLA
4 2 (0.28%) WCAG
5 1 (0.14%) GDPR
5 1 (0.14%) IFRS
5 1 (0.14%) ISO/IEC 27001
System Software
1 117 (16.20%) HDFS
2 110 (15.24%) Docker
3 13 (1.80%) Active Directory
3 13 (1.80%) Apache Flume
4 3 (0.42%) Virtual Desktop
4 3 (0.42%) Virtual Servers
5 2 (0.28%) Apache ZooKeeper
6 1 (0.14%) VMware Infrastructure
Systems Management
1 125 (17.31%) Kubernetes
2 61 (8.45%) Hadoop YARN
3 60 (8.31%) Terraform
4 45 (6.23%) Ansible
5 25 (3.46%) Oozie
6 13 (1.80%) Puppet
7 11 (1.52%) Kibana
8 10 (1.39%) Opscode Chef
9 9 (1.25%) Ambari
10 7 (0.97%) Prometheus
11 6 (0.83%) Single Sign-On
11 6 (0.83%) Zipkin
12 4 (0.55%) Grafana
12 4 (0.55%) logstash
13 3 (0.42%) Graphite
14 2 (0.28%) Netcool
15 1 (0.14%) Control-M
15 1 (0.14%) FortiGate
15 1 (0.14%) Mesos
15 1 (0.14%) Tivoli
Vendors
1 95 (13.16%) Cloudera
2 69 (9.56%) Google
3 57 (7.89%) Oracle
4 54 (7.48%) Tableau
5 50 (6.93%) Hortonworks
6 48 (6.65%) Microsoft
7 41 (5.68%) Databricks
8 30 (4.16%) Informatica
9 25 (3.46%) MapR
9 25 (3.46%) SAP
10 19 (2.63%) Talend
11 16 (2.22%) IBM
12 14 (1.94%) Splunk
13 11 (1.52%) Confluent
14 10 (1.39%) IBM Sterling
15 8 (1.11%) Red Hat
16 7 (0.97%) Ab Initio
16 7 (0.97%) Palantir
16 7 (0.97%) StreamSets
17 6 (0.83%) Sybase