Period
to 22 January 2020

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 22 January 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
22 Jan 2020
Same period 2019 Same period 2018
Rank 216 200 177
Rank change year-on-year -16 -23 +35
Contract jobs citing Apache Spark 908 1,448 1,505
As % of all contract IT jobs advertised in the UK 1.25% 1.45% 1.51%
As % of the Application Platforms category 8.71% 8.81% 9.16%
Number of daily rates quoted 659 1,022 1,115
UK median daily rate £554 £550 £550
Median daily rate % change year-on-year +0.73% - -
10th Percentile £413 £425 £400
90th Percentile £725 £723 £675
UK excluding London median daily rate £525 £525 £500
% change year-on-year - +5.00% +0.25%
Number of hourly rates quoted 1 4 10
UK median hourly rate £52.50 £50.00 £23.42
Median hourly rate % change year-on-year +5.00% +113.49% +14.24%
10th Percentile £51.25 £42.50 £16.64
90th Percentile £53.75 £140.98 £51.88
UK excluding London median hourly rate - - £19.34

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 10,427 16,430 16,438
As % of all contract IT jobs advertised in the UK 14.35% 16.49% 16.50%
Number of daily rates quoted 6,752 10,602 10,961
UK median daily rate £450 £475 £450
Median daily rate % change year-on-year -5.26% +5.56% +5.88%
10th Percentile £256 £275 £263
90th Percentile £638 £638 £623
UK excluding London median daily rate £425 £425 £400
% change year-on-year - +6.25% +3.23%
Number of hourly rates quoted 252 354 402
UK median hourly rate £21.00 £20.00 £19.42
Median hourly rate % change year-on-year +5.00% +2.99% +2.21%
10th Percentile £11.28 £12.00 £11.25
90th Percentile £52.43 £48.68 £47.50
UK excluding London median hourly rate £19.00 £18.50 £19.30
% change year-on-year +2.70% -4.12% -3.53%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 22 January 2020.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 22 January 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -22 836 £563 +2.27% 324
London -15 550 £575 - 223
UK excluding London +9 352 £525 - 125
South East -18 118 £513 -3.53% 40
North of England +18 67 £550 +4.76% 18
Scotland +54 45 £575 +15.00% 15
Yorkshire +23 38 £534 +6.75% 8
East of England +38 35 £513 -6.82% 15
Midlands +63 33 £525 +21.74% 18
South West -23 33 £550 +2.33% 10
West Midlands +74 26 £525 +31.25% 14
North West +10 21 £550 -2.65% 8
Wales +28 20 £495 -1.00% 8
North East +15 8 £650 +30.00% 2
East Midlands +21 7 £500 +11.11% 4
Northern Ireland +1 1 - - 1

For the 6 months to 22 January 2020, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 573 (63.11%) Big Data
2 563 (62.00%) Python
3 543 (59.80%) Hadoop
4 367 (40.42%) SQL
5 336 (37.00%) Kafka
6 328 (36.12%) AWS
7 318 (35.02%) Java
8 317 (34.91%) Scala
9 298 (32.82%) Agile Software Development
10 264 (29.07%) Apache Hive
11 247 (27.20%) Data Science
12 225 (24.78%) Analytics
13 219 (24.12%) Azure
14 204 (22.47%) ETL
15 192 (21.15%) Finance
16 188 (20.70%) Machine Learning
17 174 (19.16%) Apache
18 164 (18.06%) NoSQL
19 155 (17.07%) DevOps
20 153 (16.85%) Linux
21 149 (16.41%) Git (software)
22 148 (16.30%) Cloudera
23 144 (15.86%) Docker
24 141 (15.53%) PySpark
25 140 (15.42%) Data Engineering
26 135 (14.87%) Impala
27 133 (14.65%) Jenkins
27 133 (14.65%) Kubernetes
28 131 (14.43%) HDFS
28 131 (14.43%) Data Modelling

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 174 (19.16%) Apache
2 62 (6.83%) Elasticsearch
3 50 (5.51%) Jupyter
4 42 (4.63%) Apache Airflow
5 33 (3.63%) SAS
6 12 (1.32%) Luigi
7 11 (1.21%) OpenStack
8 10 (1.10%) Apache Pig
8 10 (1.10%) Confluence
9 6 (0.66%) Tomcat
10 5 (0.55%) Apache Solr
10 5 (0.55%) Cloud Foundry
10 5 (0.55%) DNN
11 4 (0.44%) GemFire
11 4 (0.44%) nginx
11 4 (0.44%) WebSphere
12 3 (0.33%) Tornado
13 2 (0.22%) CMS
13 2 (0.22%) Pachyderm
13 2 (0.22%) WebLogic
Applications
1 12 (1.32%) Microsoft Excel
2 1 (0.11%) Chatbot
2 1 (0.11%) Microsoft Office
2 1 (0.11%) Microsoft Project
Business Applications
1 9 (0.99%) FINCAD Analytics
2 4 (0.44%) SAP CRM
2 4 (0.44%) SAP ERP
2 4 (0.44%) SAP IS-U
3 3 (0.33%) Distributed Ledger
4 2 (0.22%) Adobe Campaign
4 2 (0.22%) NetSuite
5 1 (0.11%) Cognos Business Intelligence
5 1 (0.11%) SAP Oil and Gas
5 1 (0.11%) Sparx Enterprise Architect
Cloud Services
1 328 (36.12%) AWS
2 219 (24.12%) Azure
3 126 (13.88%) GCP
4 70 (7.71%) Amazon EMR
5 69 (7.60%) Amazon S3
6 64 (7.05%) Amazon Kinesis
7 55 (6.06%) Serverless
8 48 (5.29%) AWS Lambda
9 32 (3.52%) OpenShift
10 28 (3.08%) Power Platform
11 27 (2.97%) Azure Machine Learning
12 26 (2.86%) GitHub
13 25 (2.75%) Amazon SQS
14 23 (2.53%) PaaS
15 20 (2.20%) Amazon EC2
15 20 (2.20%) AWS CloudFormation
15 20 (2.20%) Azure Data Factory
15 20 (2.20%) Cloud Computing
16 14 (1.54%) Amazon EKS
17 13 (1.43%) AWS Step Functions
Communications & Networking
1 36 (3.96%) Kerberos
2 22 (2.42%) SSL
3 14 (1.54%) DNS
4 13 (1.43%) Firewall
5 10 (1.10%) SAN
6 7 (0.77%) HTTP
7 6 (0.66%) VPN
8 5 (0.55%) HTTPS
9 4 (0.44%) DSL
9 4 (0.44%) F5 BIG-IP GTM
9 4 (0.44%) F5 BIG-IP LTM
9 4 (0.44%) Internet
10 2 (0.22%) Intrusion Detection
10 2 (0.22%) RTMP
10 2 (0.22%) TCP/IP
11 1 (0.11%) Multicast
11 1 (0.11%) SSH
Database & Business Intelligence
1 573 (63.11%) Big Data
2 543 (59.80%) Hadoop
3 264 (29.07%) Apache Hive
4 164 (18.06%) NoSQL
5 135 (14.87%) Impala
6 118 (13.00%) HBase
7 98 (10.79%) Data Warehouse
8 96 (10.57%) Relational Database
9 79 (8.70%) Amazon Redshift
9 79 (8.70%) PostgreSQL
10 78 (8.59%) Data Lake
10 78 (8.59%) MongoDB
11 72 (7.93%) SQL Server
12 69 (7.60%) Spark SQL
13 64 (7.05%) Apache Cassandra
14 57 (6.28%) Amazon Athena
15 56 (6.17%) Flink
16 41 (4.52%) BigQuery
17 37 (4.07%) MySQL
18 28 (3.08%) Power BI
Development Applications
1 149 (16.41%) Git (software)
2 133 (14.65%) Jenkins
3 47 (5.18%) JIRA
4 37 (4.07%) Maven
5 28 (3.08%) Sonatype Nexus
6 25 (2.75%) Artifactory
7 18 (1.98%) Selenium
8 14 (1.54%) IntelliJ
9 13 (1.43%) Cucumber
9 13 (1.43%) Visual Studio Team System
10 12 (1.32%) Atlassian Bamboo
10 12 (1.32%) IBM UrbanCode
11 11 (1.21%) Vagrant
12 9 (0.99%) Gradle
13 8 (0.88%) Eclipse
13 8 (0.88%) JUnit
14 7 (0.77%) TeamCity
15 6 (0.66%) ERwin
15 6 (0.66%) Gerrit
15 6 (0.66%) Mockito
General
1 192 (21.15%) Finance
2 95 (10.46%) Banking
3 40 (4.41%) Front Office
4 35 (3.85%) Investment Banking
5 30 (3.30%) Retail
6 23 (2.53%) Public Sector
7 18 (1.98%) Marketing
8 17 (1.87%) Telecoms
9 14 (1.54%) Pharmaceutical
10 13 (1.43%) Legal
11 10 (1.10%) Retail Banking
12 7 (0.77%) Games
12 7 (0.77%) Manufacturing
13 2 (0.22%) Aerospace
13 2 (0.22%) Automotive
13 2 (0.22%) Corporate Banking
13 2 (0.22%) Publishing
14 1 (0.11%) Electronics
14 1 (0.11%) International Banking
14 1 (0.11%) Law
Job Titles
1 324 (35.68%) Data Engineer
2 223 (24.56%) Developer
3 124 (13.66%) Big Data Engineer
4 107 (11.78%) Data Scientist
5 98 (10.79%) Architect
6 75 (8.26%) Data Architect
7 70 (7.71%) Python Developer
8 52 (5.73%) Scala Developer
9 50 (5.51%) AWS Engineer
9 50 (5.51%) Java Developer
10 46 (5.07%) Python Engineer
11 44 (4.85%) Big Data Developer
11 44 (4.85%) DevOps Engineer
12 42 (4.63%) Senior Data Engineer
13 41 (4.52%) Senior Developer
14 36 (3.96%) Platform Engineer
14 36 (3.96%) Senior Data Scientist
15 34 (3.74%) Big Data Architect
16 32 (3.52%) Hadoop Developer
17 25 (2.75%) Apache Spark Developer
Libraries, Frameworks & Software Standards
1 336 (37.00%) Kafka
2 141 (15.53%) PySpark
3 80 (8.81%) Pandas
4 60 (6.61%) Spark Streaming
5 52 (5.73%) TensorFlow
6 49 (5.40%) Spring
7 48 (5.29%) React
8 47 (5.18%) NumPy
9 46 (5.07%) scikit-learn
10 41 (4.52%) Web Services
11 34 (3.74%) Apache NiFi
12 31 (3.41%) AngularJS
12 31 (3.41%) Natural Language Processing
13 29 (3.19%) RESTful
14 26 (2.86%) Apache Beam
14 26 (2.86%) Keras
14 26 (2.86%) Spring Boot
15 23 (2.53%) MLlib
15 23 (2.53%) Node.js
16 22 (2.42%) REST
Miscellaneous
1 96 (10.57%) Analytical Skills
2 85 (9.36%) Algorithms
3 46 (5.07%) Greenfield Project
4 45 (4.96%) Data Structures
5 26 (2.86%) Public Cloud
6 18 (1.98%) User Experience
7 16 (1.76%) Self-Motivation
8 14 (1.54%) Fixed Income
8 14 (1.54%) Foreign Exchange (FX)
8 14 (1.54%) Hedge funds
9 12 (1.32%) Credit Risk
9 12 (1.32%) Derivative
10 10 (1.10%) Management Information System
11 9 (0.99%) Grid Computing
11 9 (0.99%) Lidar
12 8 (0.88%) Data Centre
12 8 (0.88%) Smart City
13 7 (0.77%) Replication
13 7 (0.77%) Social Media
14 6 (0.66%) Market Risk
Operating Systems
1 153 (16.85%) Linux
2 44 (4.85%) Unix
3 34 (3.74%) Windows
4 23 (2.53%) Red Hat Enterprise Linux
5 22 (2.42%) CentOS
6 11 (1.21%) Windows Server
7 8 (0.88%) Ubuntu
8 5 (0.55%) VMS
9 4 (0.44%) Android
9 4 (0.44%) Apple iOS
10 2 (0.22%) SUSE Linux Enterprise Server
11 1 (0.11%) Debian
11 1 (0.11%) Mac OS X
11 1 (0.11%) Oracle Linux
11 1 (0.11%) Windows Server 2012
Processes & Methodologies
1 298 (32.82%) Agile Software Development
2 247 (27.20%) Data Science
3 225 (24.78%) Analytics
4 204 (22.47%) ETL
5 188 (20.70%) Machine Learning
6 155 (17.07%) DevOps
7 140 (15.42%) Data Engineering
8 131 (14.43%) Data Modelling
9 117 (12.89%) Computer Science
10 99 (10.90%) Continuous Integration
11 94 (10.35%) Visualisation
12 93 (10.24%) Software Engineering
13 85 (9.36%) Scrum
14 84 (9.25%) Artificial Intelligence
14 84 (9.25%) Migration
15 83 (9.14%) TDD
16 82 (9.03%) Microservices
17 78 (8.59%) Problem-Solving
18 76 (8.37%) Business Intelligence
19 72 (7.93%) Data Analysis
Programming Languages
1 563 (62.00%) Python
2 367 (40.42%) SQL
3 318 (35.02%) Java
4 317 (34.91%) Scala
5 94 (10.35%) R
6 42 (4.63%) Bash Shell
7 39 (4.30%) JavaScript
8 37 (4.07%) Go
9 34 (3.74%) C#
10 28 (3.08%) C
11 27 (2.97%) C++
12 26 (2.86%) Ruby
13 21 (2.31%) Shell Script
14 19 (2.09%) Perl
15 17 (1.87%) Kotlin
16 14 (1.54%) PowerShell
17 11 (1.21%) Groovy
18 9 (0.99%) Java 8
19 8 (0.88%) Clojure
19 8 (0.88%) PL/SQL
Qualifications
1 115 (12.67%) Degree
2 37 (4.07%) PhD
3 35 (3.85%) Security Cleared
4 28 (3.08%) Computer Science Degree
5 25 (2.75%) Master's Degree
6 23 (2.53%) SC Cleared
7 20 (2.20%) Numerate Degree
8 8 (0.88%) AWS Certification
9 6 (0.66%) BPSS Clearance
9 6 (0.66%) Cisco Certification
9 6 (0.66%) DBS Check
10 4 (0.44%) CCIE
10 4 (0.44%) CCNP
11 3 (0.33%) Microsoft Certification
11 3 (0.33%) PMI Certification
11 3 (0.33%) PMP
12 2 (0.22%) CCNA
12 2 (0.22%) CISSP
12 2 (0.22%) Postgraduate
12 2 (0.22%) SSCP
Quality Assurance & Compliance
1 23 (2.53%) Data Quality
2 22 (2.42%) QA
3 5 (0.55%) GDPR
3 5 (0.55%) MiFID
4 4 (0.44%) Disclosure Scotland
4 4 (0.44%) EMIR
4 4 (0.44%) PCI DSS
5 2 (0.22%) WCAG
6 1 (0.11%) ISO/IEC 27001
System Software
1 144 (15.86%) Docker
2 131 (14.43%) HDFS
3 17 (1.87%) Active Directory
4 10 (1.10%) Virtual Desktop
4 10 (1.10%) Virtual Servers
5 9 (0.99%) VirtualBox
6 8 (0.88%) Apache Flume
7 5 (0.55%) Apache ZooKeeper
8 2 (0.22%) VMware Infrastructure
Systems Management
1 133 (14.65%) Kubernetes
2 81 (8.92%) Hadoop YARN
3 77 (8.48%) Ansible
3 77 (8.48%) Terraform
4 37 (4.07%) Oozie
5 26 (2.86%) Puppet
6 16 (1.76%) Opscode Chef
7 10 (1.10%) Kibana
8 9 (0.99%) Ambari
8 9 (0.99%) Docker Swarm
9 6 (0.66%) logstash
9 6 (0.66%) Single Sign-On
10 4 (0.44%) FortiGate
10 4 (0.44%) Prometheus
11 3 (0.33%) Cobbler
11 3 (0.33%) Control-M
11 3 (0.33%) Graphite
11 3 (0.33%) Mesos
11 3 (0.33%) Red Hat Satellite
12 2 (0.22%) Salt
Vendors
1 148 (16.30%) Cloudera
2 90 (9.91%) Hortonworks
3 88 (9.69%) Oracle
4 84 (9.25%) Google
5 72 (7.93%) Microsoft
6 67 (7.38%) Tableau
7 61 (6.72%) Databricks
8 37 (4.07%) Informatica
9 32 (3.52%) Talend
10 25 (2.75%) Red Hat
11 22 (2.42%) IBM
11 22 (2.42%) Palantir
11 22 (2.42%) SAP
12 20 (2.20%) Teradata
13 19 (2.09%) MapR
13 19 (2.09%) StreamSets
14 17 (1.87%) Splunk
15 9 (0.99%) Ab Initio
15 9 (0.99%) Bloomberg
15 9 (0.99%) Datactics