Period
to 14 October 2019

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 14 October 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
14 Oct 2019
Same period 2018 Same period 2017
Rank 166 195 193
Rank change year-on-year +29 -2 +96
Contract jobs citing Apache Spark 1,396 1,461 1,372
As % of all contract IT jobs advertised in the UK 1.63% 1.43% 1.39%
As % of the Application Platforms category 10.91% 8.56% 8.32%
Number of daily rates quoted 1,025 1,025 981
UK median daily rate £550 £550 £550
10th Percentile £425 £425 £400
90th Percentile £713 £700 £688
UK excluding London median daily rate £525 £505 £475
% change year-on-year +3.96% +6.32% +8.57%
Number of hourly rates quoted 3 10 6
UK median hourly rate £24.65 £45.00 £19.34
Median hourly rate % change year-on-year -45.22% +132.68% -5.66%
10th Percentile £21.93 £16.83 £16.64
90th Percentile £32.93 £84.05 £41.25
UK excluding London median hourly rate £24.65 £22.00 £19.34
% change year-on-year +12.05% +13.75% -5.66%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 12,792 17,063 16,486
As % of all contract IT jobs advertised in the UK 14.98% 16.69% 16.75%
Number of daily rates quoted 8,243 11,101 10,740
UK median daily rate £455 £450 £440
Median daily rate % change year-on-year +1.11% +2.27% +4.76%
10th Percentile £263 £272 £263
90th Percentile £631 £625 £600
UK excluding London median daily rate £425 £425 £400
% change year-on-year - +6.25% +2.56%
Number of hourly rates quoted 313 416 405
UK median hourly rate £21.50 £21.25 £20.00
Median hourly rate % change year-on-year +1.18% +6.25% -
10th Percentile £11.05 £12.00 £11.00
90th Percentile £51.85 £51.25 £48.75
UK excluding London median hourly rate £20.91 £20.75 £20.00
% change year-on-year +0.75% +3.75% -3.75%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 14 October 2019.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 14 October 2019.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 14 October 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +25 1,304 £555 +0.91% 242
London +36 960 £575 - 177
UK excluding London +2 421 £525 +3.96% 83
South East -26 121 £530 +6.00% 21
North of England +37 102 £539 +2.62% 19
Scotland +60 77 £490 +8.89% 18
North West +21 55 £525 -2.33% 8
East of England +36 54 £525 -4.55% 7
Yorkshire +25 40 £538 +13.16% 8
Midlands +60 32 £513 +20.65% 5
South West -42 30 £600 +14.29% 13
West Midlands +50 19 £470 +25.33% 2
East Midlands +29 13 £525 +23.53% 3
North East +24 7 £563 -10.00% 3
Wales +21 3 £700 +35.92%
Northern Ireland -8 2 - -

For the 6 months to 14 October 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 880 (63.04%) Big Data
2 860 (61.60%) Python
3 837 (59.96%) Hadoop
4 572 (40.97%) SQL
5 528 (37.82%) Kafka
6 497 (35.60%) Agile Software Development
7 493 (35.32%) AWS
8 462 (33.09%) Java
9 460 (32.95%) Scala
10 452 (32.38%) Apache Hive
11 389 (27.87%) Data Science
12 336 (24.07%) Analytics
13 328 (23.50%) Finance
14 317 (22.71%) Azure
15 308 (22.06%) DevOps
16 299 (21.42%) ETL
17 272 (19.48%) Machine Learning
18 269 (19.27%) Apache
19 260 (18.62%) Docker
20 248 (17.77%) Linux
21 241 (17.26%) GCP
21 241 (17.26%) Data Engineering
22 240 (17.19%) Data Modelling
23 233 (16.69%) Data Warehouse
23 233 (16.69%) NoSQL
24 221 (15.83%) Git (software)
25 208 (14.90%) Impala
26 201 (14.40%) Jenkins
27 192 (13.75%) Kubernetes
28 183 (13.11%) Cloudera

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 269 (19.27%) Apache
2 92 (6.59%) Elasticsearch
3 78 (5.59%) Apache Airflow
4 48 (3.44%) Jupyter
5 38 (2.72%) SAS
6 25 (1.79%) Apache Pig
7 24 (1.72%) Confluence
8 23 (1.65%) Apache Solr
9 14 (1.00%) Luigi
10 10 (0.72%) Pachyderm
10 10 (0.72%) Tomcat
11 9 (0.64%) Jetty
12 7 (0.50%) Oracle Coherence
13 6 (0.43%) nginx
14 5 (0.36%) Cloud Foundry
14 5 (0.36%) JBoss
14 5 (0.36%) Oracle Self-Service HR
14 5 (0.36%) WebLogic
15 4 (0.29%) ColdFusion
15 4 (0.29%) SharePoint
Applications
1 34 (2.44%) Microsoft Excel
2 5 (0.36%) Microsoft Office
3 1 (0.072%) Stata
Business Applications
1 6 (0.43%) FINCAD Analytics
1 6 (0.43%) SAP Oil and Gas
2 2 (0.14%) Adobe Campaign
2 2 (0.14%) Cognos Business Intelligence
3 1 (0.072%) Distributed Ledger
Cloud Services
1 493 (35.32%) AWS
2 317 (22.71%) Azure
3 241 (17.26%) GCP
4 134 (9.60%) Amazon S3
5 125 (8.95%) Serverless
6 124 (8.88%) Amazon EMR
7 103 (7.38%) AWS Lambda
8 95 (6.81%) Amazon Kinesis
9 61 (4.37%) Azure Data Factory
10 56 (4.01%) Power Platform
11 45 (3.22%) GitHub
12 42 (3.01%) Amazon SQS
13 41 (2.94%) Amazon EC2
14 28 (2.01%) CloudFront
14 28 (2.01%) OpenShift
15 24 (1.72%) AWS CloudFormation
16 23 (1.65%) Cloud Computing
17 21 (1.50%) PaaS
18 18 (1.29%) Amazon SageMaker
18 18 (1.29%) Azure Machine Learning
Communications & Networking
1 29 (2.08%) Kerberos
2 16 (1.15%) SSL
3 15 (1.07%) DNS
4 12 (0.86%) HTTP
5 8 (0.57%) Firewall
5 8 (0.57%) TCP/IP
6 7 (0.50%) Internet
6 7 (0.50%) SAN
7 5 (0.36%) SSH
8 4 (0.29%) DSL
8 4 (0.29%) F5 BIG-IP GTM
8 4 (0.29%) F5 BIG-IP LTM
8 4 (0.29%) FTP
9 3 (0.21%) Software-Defined Networking
10 2 (0.14%) Intrusion Detection
10 2 (0.14%) Multicast
11 1 (0.072%) LAN
11 1 (0.072%) VPN
11 1 (0.072%) WAN
Database & Business Intelligence
1 880 (63.04%) Big Data
2 837 (59.96%) Hadoop
3 452 (32.38%) Apache Hive
4 233 (16.69%) Data Warehouse
4 233 (16.69%) NoSQL
5 208 (14.90%) Impala
6 160 (11.46%) HBase
7 150 (10.74%) Data Lake
8 128 (9.17%) PostgreSQL
9 124 (8.88%) MongoDB
10 113 (8.09%) Amazon Redshift
11 98 (7.02%) Relational Database
12 93 (6.66%) SQL Server
13 92 (6.59%) Apache Cassandra
14 85 (6.09%) Spark SQL
15 67 (4.80%) BigQuery
16 64 (4.58%) MySQL
17 56 (4.01%) Power BI
18 51 (3.65%) Parquet
19 49 (3.51%) Flink
Development Applications
1 221 (15.83%) Git (software)
2 201 (14.40%) Jenkins
3 67 (4.80%) JIRA
4 46 (3.30%) Atlassian Bamboo
5 40 (2.87%) Maven
6 34 (2.44%) Sonatype Nexus
7 33 (2.36%) Artifactory
8 29 (2.08%) IntelliJ
9 22 (1.58%) GitLab
10 21 (1.50%) Gradle
11 20 (1.43%) TeamCity
12 19 (1.36%) Eclipse
13 17 (1.22%) Vagrant
14 16 (1.15%) Cucumber
14 16 (1.15%) Selenium
15 14 (1.00%) Subversion
16 13 (0.93%) CircleCI
16 13 (0.93%) JUnit
17 11 (0.79%) Travis CI
17 11 (0.79%) Visual Studio Team System
General
1 328 (23.50%) Finance
2 158 (11.32%) Banking
3 61 (4.37%) Investment Banking
4 59 (4.23%) Front Office
5 40 (2.87%) Retail
6 36 (2.58%) Legal
7 24 (1.72%) Public Sector
8 23 (1.65%) Retail Banking
9 17 (1.22%) Marketing
9 17 (1.22%) Telecoms
10 13 (0.93%) Pharmaceutical
11 11 (0.79%) Financial Institution
12 10 (0.72%) Advertising
12 10 (0.72%) Publishing
13 8 (0.57%) Games
14 7 (0.50%) Manufacturing
15 4 (0.29%) Aerospace
16 2 (0.14%) Automotive
16 2 (0.14%) Back Office
16 2 (0.14%) International Banking
Job Titles
1 546 (39.11%) Data Engineer
2 350 (25.07%) Developer
3 197 (14.11%) Big Data Engineer
4 135 (9.67%) Architect
5 131 (9.38%) Python Engineer
6 99 (7.09%) Big Data Developer
7 98 (7.02%) Data Scientist
8 96 (6.88%) Python Developer
9 94 (6.73%) AWS Engineer
10 93 (6.66%) Java Developer
11 91 (6.52%) Data Architect
12 73 (5.23%) Scala Developer
13 71 (5.09%) DevOps Engineer
14 69 (4.94%) Hadoop Engineer
15 60 (4.30%) Senior Data Engineer
16 55 (3.94%) Senior Developer
17 53 (3.80%) Hadoop Developer
18 52 (3.72%) Cloud Engineer
19 45 (3.22%) Apache Spark Developer
20 42 (3.01%) Platform Engineer
Libraries, Frameworks & Software Standards
1 528 (37.82%) Kafka
2 173 (12.39%) PySpark
3 89 (6.38%) Pandas
4 86 (6.16%) Spring
5 75 (5.37%) Spark Streaming
6 63 (4.51%) Web Services
7 54 (3.87%) scikit-learn
8 53 (3.80%) Natural Language Processing
9 48 (3.44%) Spring Boot
10 45 (3.22%) NumPy
10 45 (3.22%) React
10 45 (3.22%) TensorFlow
11 39 (2.79%) RESTful
12 38 (2.72%) Apache NiFi
13 36 (2.58%) JSON
14 35 (2.51%) RabbitMQ
14 35 (2.51%) REST
15 28 (2.01%) Apache Avro
16 27 (1.93%) Node.js
17 26 (1.86%) AngularJS
Miscellaneous
1 128 (9.17%) Analytical Skills
2 120 (8.60%) Algorithms
3 95 (6.81%) Data Structures
4 75 (5.37%) Greenfield Project
5 36 (2.58%) Self-Motivation
6 33 (2.36%) User Experience
7 27 (1.93%) Public Cloud
8 23 (1.65%) Cloud Native
9 19 (1.36%) Derivative
10 16 (1.15%) Foreign Exchange (FX)
10 16 (1.15%) Grid Computing
11 15 (1.07%) Hedge funds
11 15 (1.07%) Linux Command Line
12 14 (1.00%) Fixed Income
13 12 (0.86%) Replication
14 11 (0.79%) Life Science
14 11 (0.79%) Market Risk
15 9 (0.64%) Enterprise Software
16 8 (0.57%) Credit Risk
16 8 (0.57%) Lidar
Operating Systems
1 248 (17.77%) Linux
2 109 (7.81%) Unix
3 68 (4.87%) Windows
4 34 (2.44%) Red Hat Enterprise Linux
5 22 (1.58%) CentOS
6 11 (0.79%) Ubuntu
7 9 (0.64%) Windows Server
8 7 (0.50%) Android
8 7 (0.50%) Apple iOS
8 7 (0.50%) VMS
9 5 (0.36%) Debian
9 5 (0.36%) Mac OS X
10 2 (0.14%) SUSE Linux Enterprise Server
11 1 (0.072%) Oracle Linux
11 1 (0.072%) Windows Server 2012
Processes & Methodologies
1 497 (35.60%) Agile Software Development
2 389 (27.87%) Data Science
3 336 (24.07%) Analytics
4 308 (22.06%) DevOps
5 299 (21.42%) ETL
6 272 (19.48%) Machine Learning
7 241 (17.26%) Data Engineering
8 240 (17.19%) Data Modelling
9 171 (12.25%) Computer Science
10 151 (10.82%) Scrum
11 137 (9.81%) Continuous Integration
12 136 (9.74%) TDD
13 132 (9.46%) Artificial Intelligence
14 131 (9.38%) Software Engineering
15 127 (9.10%) Migration
16 123 (8.81%) Problem-Solving
17 112 (8.02%) Business Intelligence
18 110 (7.88%) Microservices
19 101 (7.23%) Visualisation
20 96 (6.88%) Data Analysis
Programming Languages
1 860 (61.60%) Python
2 572 (40.97%) SQL
3 462 (33.09%) Java
4 460 (32.95%) Scala
5 149 (10.67%) R
6 77 (5.52%) JavaScript
7 46 (3.30%) Bash Shell
7 46 (3.30%) C#
8 39 (2.79%) Go
9 37 (2.65%) Shell Script
10 35 (2.51%) C++
11 33 (2.36%) Ruby
12 26 (1.86%) Perl
13 25 (1.79%) PL/SQL
14 23 (1.65%) Java 8
15 22 (1.58%) T-SQL
16 20 (1.43%) C
16 20 (1.43%) Kotlin
17 19 (1.36%) Clojure
18 17 (1.22%) Groovy
Qualifications
1 138 (9.89%) Degree
2 58 (4.15%) Security Cleared
3 54 (3.87%) PhD
4 46 (3.30%) Computer Science Degree
5 44 (3.15%) SC Cleared
6 31 (2.22%) Master's Degree
7 30 (2.15%) AWS Certification
8 18 (1.29%) PMI Certification
8 18 (1.29%) PMP
9 14 (1.00%) Numerate Degree
10 8 (0.57%) BPSS Clearance
11 7 (0.50%) DV Cleared
11 7 (0.50%) TOGAF Certification
12 6 (0.43%) DBS Check
13 5 (0.36%) ISTQB
14 4 (0.29%) CCIE
14 4 (0.29%) CCNP
14 4 (0.29%) Cisco Certification
15 2 (0.14%) HNC
15 2 (0.14%) HND
Quality Assurance & Compliance
1 42 (3.01%) Data Quality
2 19 (1.36%) QA
3 18 (1.29%) GDPR
4 7 (0.50%) MiFID
5 6 (0.43%) EMIR
6 4 (0.29%) Disclosure Scotland
7 1 (0.072%) ISO/IEC 27001
7 1 (0.072%) PCI DSS
System Software
1 260 (18.62%) Docker
2 158 (11.32%) HDFS
3 41 (2.94%) Apache Flume
4 18 (1.29%) Active Directory
5 16 (1.15%) Apache ZooKeeper
6 9 (0.64%) VirtualBox
7 7 (0.50%) Virtual Desktop
7 7 (0.50%) Virtual Servers
8 2 (0.14%) VMware Infrastructure
9 1 (0.072%) Virtual Machines
Systems Management
1 192 (13.75%) Kubernetes
2 116 (8.31%) Ansible
3 103 (7.38%) Terraform
4 92 (6.59%) Hadoop YARN
5 73 (5.23%) Oozie
6 57 (4.08%) Puppet
7 34 (2.44%) Opscode Chef
8 13 (0.93%) Kibana
9 11 (0.79%) Docker Swarm
10 10 (0.72%) Kubeflow
10 10 (0.72%) logstash
11 8 (0.57%) Salt
12 7 (0.50%) Control-M
13 6 (0.43%) Grafana
13 6 (0.43%) Mesos
14 5 (0.36%) Ambari
14 5 (0.36%) Prometheus
15 4 (0.29%) Cobbler
15 4 (0.29%) FortiGate
15 4 (0.29%) Red Hat Satellite
Vendors
1 183 (13.11%) Cloudera
2 175 (12.54%) Google
3 141 (10.10%) Oracle
4 116 (8.31%) Databricks
5 97 (6.95%) Informatica
6 87 (6.23%) Tableau
7 85 (6.09%) Hortonworks
8 81 (5.80%) Microsoft
9 68 (4.87%) Talend
10 44 (3.15%) SAP
11 39 (2.79%) IBM
12 36 (2.58%) Red Hat
13 31 (2.22%) Teradata
14 30 (2.15%) Splunk
15 28 (2.01%) IBM Sterling
16 19 (1.36%) MapR
16 19 (1.36%) StreamSets
17 17 (1.22%) Palantir
18 14 (1.00%) MicroStrategy
19 13 (0.93%) Sybase