Period
to 13 July 2020

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 13 July 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
13 Jul 2020
Same period 2019 Same period 2018
Rank 187 158 182
Rank change year-on-year -29 +24 +36
Contract jobs citing Apache Spark 561 1,651 1,589
As % of all contract jobs advertised in the UK 1.38% 1.76% 1.52%
As % of the Application Platforms category 9.38% 11.17% 9.20%
Number of daily rates quoted 381 1,224 1,148
Median daily rate £530 £550 £550
Median daily rate % change year-on-year -3.64% - -
10th Percentile £409 £439 £416
90th Percentile £688 £700 £688
UK excluding London median daily rate £500 £525 £500
% change year-on-year -4.76% +5.00% +5.26%
Number of hourly rates quoted 11 1 8
Median hourly rate £35.00 £35.00 £35.00
UK excluding London median hourly rate £35.00 £35.00 £22.25
% change year-on-year - +57.30% -36.43%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 5,981 14,783 17,281
As % of all contract IT jobs advertised in the UK 14.76% 15.72% 16.52%
Number of daily rates quoted 3,719 9,406 11,343
Median daily rate £475 £475 £450
Median daily rate % change year-on-year - +5.56% +4.65%
10th Percentile £300 £275 £263
90th Percentile £650 £638 £625
UK excluding London median daily rate £450 £425 £415
% change year-on-year +5.88% +2.41% +3.75%
Number of hourly rates quoted 157 334 399
Median hourly rate £24.00 £20.86 £20.00
Median hourly rate % change year-on-year +15.08% +4.28% +1.73%
10th Percentile £12.94 £12.00 £11.83
90th Percentile £49.20 £51.58 £50.25
Median hourly rate £20.00 £20.00 £19.00
% change year-on-year - +5.26% -2.56%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 13 July 2020.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 13 July 2020.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 15 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 13 July 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -35 506 £540 -1.82% 100
London -24 381 £550 -3.51% 73
UK excluding London -28 164 £500 -4.76% 33
South East +65 56 £475 -9.52% 8
Wales +61 31 £500 -4.76% 3
Work from Home -18 29 £539 -13.76% 14
North of England -21 29 £538 +4.88% 6
East of England +34 16 £656 +25.00% 5
North West +21 16 £550 - 4
Scotland -7 16 £525 +10.53% 3
South West +65 15 £528 +0.48% 3
Yorkshire +5 12 £475 -5.00% 2
Midlands +64 3 £550 +7.21% 5
West Midlands +59 2 £550 +10.00% 3
East Midlands +47 1 - - 2

For the 6 months to 13 July 2020, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 354 (63.10%) Python
2 350 (62.39%) Big Data
3 300 (53.48%) Hadoop
4 230 (41.00%) SQL
5 229 (40.82%) Scala
6 213 (37.97%) AWS
7 207 (36.90%) Kafka
8 201 (35.83%) Agile Software Development
9 200 (35.65%) Java
10 161 (28.70%) Apache Hive
11 154 (27.45%) Analytics
12 128 (22.82%) Azure
13 126 (22.46%) GCP
14 123 (21.93%) Machine Learning
15 120 (21.39%) NoSQL
16 118 (21.03%) Data Science
17 112 (19.96%) ETL
18 105 (18.72%) Kubernetes
19 104 (18.54%) Finance
20 99 (17.65%) PySpark
21 92 (16.40%) DevOps
22 91 (16.22%) Apache
23 90 (16.04%) Data Modelling
24 88 (15.69%) Git (software)
25 85 (15.15%) Cloudera
26 84 (14.97%) Docker
27 80 (14.26%) Computer Science
28 78 (13.90%) Business Intelligence
29 77 (13.73%) Microservices
30 76 (13.55%) HDFS

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 91 (16.22%) Apache
2 33 (5.88%) Elasticsearch
3 26 (4.63%) Apache Airflow
4 22 (3.92%) Apache Pig
5 17 (3.03%) SAS
6 13 (2.32%) Confluence
6 13 (2.32%) Jupyter
7 8 (1.43%) Tomcat
8 6 (1.07%) nginx
8 6 (1.07%) Tornado
9 3 (0.53%) Apache Pulsar
9 3 (0.53%) Cloud Foundry
9 3 (0.53%) GemFire
9 3 (0.53%) Luigi
9 3 (0.53%) Oracle Workflow
9 3 (0.53%) WebSphere
10 2 (0.36%) Apache Solr
10 2 (0.36%) IIS
10 2 (0.36%) Jetty
10 2 (0.36%) WebLogic
Applications
1 4 (0.71%) MS Visio
2 1 (0.18%) Chatbot
2 1 (0.18%) Microsoft Excel
2 1 (0.18%) Microsoft PowerPoint
2 1 (0.18%) SPSS
Business Applications
1 4 (0.71%) SAP Oil and Gas
2 1 (0.18%) FINCAD Analytics
2 1 (0.18%) SAP CRM
2 1 (0.18%) SAP ERP
2 1 (0.18%) SAP IS-U
2 1 (0.18%) Shopify
2 1 (0.18%) Sparx Enterprise Architect
Cloud Services
1 213 (37.97%) AWS
2 128 (22.82%) Azure
3 126 (22.46%) GCP
4 49 (8.73%) Amazon S3
5 38 (6.77%) Serverless
6 33 (5.88%) Amazon Kinesis
7 31 (5.53%) Amazon EMR
8 30 (5.35%) AWS Lambda
9 23 (4.10%) Amazon EC2
10 20 (3.57%) Power Platform
11 19 (3.39%) Azure Data Factory
11 19 (3.39%) GitHub
12 17 (3.03%) Cloud Computing
13 15 (2.67%) Amazon SQS
14 11 (1.96%) SaaS
15 7 (1.25%) Amazon Glacier
15 7 (1.25%) AWS CloudFormation
15 7 (1.25%) Azure Active Directory
15 7 (1.25%) CloudFront
15 7 (1.25%) Virtual Private Cloud
Communications & Networking
1 27 (4.81%) SSL
2 26 (4.63%) HTTPS
3 12 (2.14%) HTTP
4 9 (1.60%) Kerberos
5 7 (1.25%) RTMP
6 5 (0.89%) Internet
7 3 (0.53%) DNS
7 3 (0.53%) DSL
7 3 (0.53%) Firewall
7 3 (0.53%) SAN
7 3 (0.53%) Skype
7 3 (0.53%) VPN
8 2 (0.36%) 5G
8 2 (0.36%) TCP/IP
9 1 (0.18%) MQTT
9 1 (0.18%) Wi-Fi
Database & Business Intelligence
1 350 (62.39%) Big Data
2 300 (53.48%) Hadoop
3 161 (28.70%) Apache Hive
4 120 (21.39%) NoSQL
5 76 (13.55%) Impala
6 69 (12.30%) HBase
7 61 (10.87%) Data Lake
8 53 (9.45%) MongoDB
9 52 (9.27%) Apache Cassandra
10 51 (9.09%) Data Warehouse
11 50 (8.91%) Relational Database
12 42 (7.49%) Amazon Redshift
12 42 (7.49%) SQL Server
13 30 (5.35%) BigQuery
14 25 (4.46%) Data Mining
14 25 (4.46%) Sqoop
15 24 (4.28%) Spark SQL
16 23 (4.10%) PostgreSQL
17 22 (3.92%) Amazon RDS
18 20 (3.57%) Power BI
Development Applications
1 88 (15.69%) Git (software)
2 61 (10.87%) Jenkins
3 25 (4.46%) JIRA
4 22 (3.92%) Maven
5 19 (3.39%) TeamCity
6 16 (2.85%) Sonatype Nexus
7 14 (2.50%) GitLab
8 11 (1.96%) Cucumber
9 10 (1.78%) Gradle
9 10 (1.78%) Visual Studio Team System
10 9 (1.60%) JUnit
10 9 (1.60%) Selenium
11 8 (1.43%) Artifactory
12 7 (1.25%) IBM UrbanCode
12 7 (1.25%) Mockito
12 7 (1.25%) NUnit
13 6 (1.07%) CircleCI
13 6 (1.07%) Hudson
13 6 (1.07%) ProC
13 6 (1.07%) Team Foundation Server
General
1 104 (18.54%) Finance
2 75 (13.37%) Banking
3 31 (5.53%) Retail
4 26 (4.63%) Investment Banking
5 21 (3.74%) Front Office
6 20 (3.57%) Telecoms
7 18 (3.21%) Marketing
8 15 (2.67%) Public Sector
9 14 (2.50%) Legal
10 11 (1.96%) Pharmaceutical
11 6 (1.07%) Financial Institution
12 5 (0.89%) Games
13 3 (0.53%) Manufacturing
14 2 (0.36%) Electronics
14 2 (0.36%) Law
14 2 (0.36%) Retail Banking
15 1 (0.18%) Aerospace
15 1 (0.18%) Automotive
15 1 (0.18%) Dutch Language
15 1 (0.18%) Publishing
Job Titles
1 209 (37.25%) Data Engineer
2 140 (24.96%) Developer
3 84 (14.97%) Big Data Engineer
4 68 (12.12%) Data Scientist
5 57 (10.16%) Architect
6 54 (9.63%) Java Developer
7 50 (8.91%) Python Engineer
8 40 (7.13%) AWS Engineer
9 34 (6.06%) Data Architect
10 30 (5.35%) DevOps Engineer
10 30 (5.35%) Scala Developer
11 27 (4.81%) Hadoop Engineer
12 25 (4.46%) Big Data Developer
12 25 (4.46%) Senior Data Scientist
13 23 (4.10%) Azure Engineer
14 20 (3.57%) Python Developer
15 19 (3.39%) Senior Developer
16 15 (2.67%) Cloud Engineer
16 15 (2.67%) Software Engineer
17 14 (2.50%) Full Stack Developer
Libraries, Frameworks & Software Standards
1 207 (36.90%) Kafka
2 99 (17.65%) PySpark
3 57 (10.16%) Spring
4 51 (9.09%) TensorFlow
5 40 (7.13%) RESTful
6 31 (5.53%) Spark Streaming
7 30 (5.35%) scikit-learn
8 28 (4.99%) Pandas
8 28 (4.99%) Spring Boot
9 25 (4.46%) Spring Security
10 24 (4.28%) JAAS
11 23 (4.10%) Akka
12 22 (3.92%) .NET
12 22 (3.92%) AngularJS
12 22 (3.92%) MLlib
12 22 (3.92%) React
12 22 (3.92%) Web Services
13 20 (3.57%) Guice
14 19 (3.39%) HTML
15 18 (3.21%) Natural Language Processing
Miscellaneous
1 63 (11.23%) Analytical Skills
2 50 (8.91%) Algorithms
3 17 (3.03%) Data Structures
4 15 (2.67%) Greenfield Project
5 14 (2.50%) Credit Risk
6 13 (2.32%) Foreign Exchange (FX)
7 12 (2.14%) Management Information System
7 12 (2.14%) Market Risk
8 11 (1.96%) Enterprise Software
9 10 (1.78%) Cloud Native
9 10 (1.78%) Self-Motivation
9 10 (1.78%) Virtual Team
10 9 (1.60%) FMCG
11 8 (1.43%) Life Science
11 8 (1.43%) PMI
12 7 (1.25%) Analytical Mindset
12 7 (1.25%) Reference Data
13 6 (1.07%) Social Media
14 5 (0.89%) Data Centre
15 4 (0.71%) Grid Computing
Operating Systems
1 71 (12.66%) Linux
2 17 (3.03%) Unix
3 14 (2.50%) Windows
4 12 (2.14%) Red Hat Enterprise Linux
5 7 (1.25%) CentOS
5 7 (1.25%) Debian
6 5 (0.89%) Ubuntu
7 3 (0.53%) Windows Server
8 1 (0.18%) DC/OS
Processes & Methodologies
1 201 (35.83%) Agile Software Development
2 154 (27.45%) Analytics
3 123 (21.93%) Machine Learning
4 118 (21.03%) Data Science
5 112 (19.96%) ETL
6 92 (16.40%) DevOps
7 90 (16.04%) Data Modelling
8 80 (14.26%) Computer Science
9 78 (13.90%) Business Intelligence
10 77 (13.73%) Microservices
11 76 (13.55%) Data Engineering
12 61 (10.87%) Software Engineering
13 60 (10.70%) TDD
14 59 (10.52%) Continuous Integration
15 56 (9.98%) Visualisation
16 55 (9.80%) Mathematics
17 49 (8.73%) Scrum
18 47 (8.38%) Artificial Intelligence
19 45 (8.02%) Test Automation
20 44 (7.84%) Use Case
Programming Languages
1 354 (63.10%) Python
2 230 (41.00%) SQL
3 229 (40.82%) Scala
4 200 (35.65%) Java
5 50 (8.91%) JavaScript
6 41 (7.31%) R
7 34 (6.06%) C#
8 31 (5.53%) Java 8
9 30 (5.35%) Ruby
10 26 (4.63%) Kotlin
11 23 (4.10%) Go
12 18 (3.21%) Bash Shell
13 16 (2.85%) T-SQL
14 13 (2.32%) Shell Script
15 11 (1.96%) Groovy
15 11 (1.96%) PHP
16 10 (1.78%) PowerShell
17 8 (1.43%) C++
17 8 (1.43%) PL/SQL
18 7 (1.25%) TypeScript
Qualifications
1 55 (9.80%) Degree
1 55 (9.80%) Security Cleared
2 48 (8.56%) SC Cleared
3 23 (4.10%) Computer Science Degree
4 14 (2.50%) Master's Degree
5 12 (2.14%) Numerate Degree
6 9 (1.60%) PhD
7 8 (1.43%) PMI Certification
8 7 (1.25%) AWS Certification
9 4 (0.71%) Microsoft Certification
9 4 (0.71%) Postgraduate
10 3 (0.53%) DBS Check
11 1 (0.18%) BPSS Clearance
11 1 (0.18%) DV Cleared
11 1 (0.18%) MBA
Quality Assurance & Compliance
1 9 (1.60%) Data Quality
2 6 (1.07%) SLA
3 5 (0.89%) MiFID
3 5 (0.89%) QA
4 4 (0.71%) GxP
5 1 (0.18%) CDISC
5 1 (0.18%) IFRS
5 1 (0.18%) ISO/IEC 27001
5 1 (0.18%) WCAG
System Software
1 84 (14.97%) Docker
2 76 (13.55%) HDFS
3 16 (2.85%) Apache Flume
4 7 (1.25%) Active Directory
5 5 (0.89%) Apache ZooKeeper
6 3 (0.53%) Virtual Desktop
6 3 (0.53%) Virtual Servers
Systems Management
1 105 (18.72%) Kubernetes
2 46 (8.20%) Terraform
3 29 (5.17%) Ansible
4 26 (4.63%) Hadoop YARN
5 24 (4.28%) Oozie
6 22 (3.92%) Prometheus
6 22 (3.92%) Zipkin
7 11 (1.96%) Puppet
8 6 (1.07%) Kibana
8 6 (1.07%) Opscode Chef
9 5 (0.89%) Ambari
10 4 (0.71%) Grafana
11 3 (0.53%) Single Sign-On
12 2 (0.36%) Fluentd
12 2 (0.36%) logstash
12 2 (0.36%) Netcool
13 1 (0.18%) Docker Swarm
13 1 (0.18%) FortiGate
13 1 (0.18%) Marathon
13 1 (0.18%) Mesos
Vendors
1 85 (15.15%) Cloudera
2 45 (8.02%) Google
3 42 (7.49%) Tableau
4 40 (7.13%) Oracle
5 35 (6.24%) Databricks
5 35 (6.24%) Hortonworks
6 29 (5.17%) Talend
7 21 (3.74%) Informatica
8 20 (3.57%) Microsoft
9 19 (3.39%) MapR
10 18 (3.21%) SAP
11 12 (2.14%) IBM
11 12 (2.14%) StreamSets
11 12 (2.14%) Teradata
12 11 (1.96%) Red Hat
13 7 (1.25%) Confluent
13 7 (1.25%) MuleSoft
13 7 (1.25%) Splunk
14 6 (1.07%) IBM Sterling
14 6 (1.07%) Salesforce.com