Period
to 20 June 2019

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 20 June 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
20 Jun 2019
Same period 2018 Same period 2017
Rank 152 173 226
Rank change year-on-year +21 +53 +202
Contract jobs citing Apache Spark 1,645 1,569 1,083
As % of all contract IT jobs advertised in the UK 1.81% 1.57% 1.19%
As % of the Application Platforms category 11.53% 9.61% 6.97%
Number of daily rates quoted 1,182 1,126 767
UK median daily rate £550 £550 £550
10th Percentile £438 £415 £400
90th Percentile £710 £688 £688
UK excluding London median daily rate £513 £500 £475
% change year-on-year +2.60% +5.26% +11.76%
Number of hourly rates quoted 1 8 0
UK median hourly rate £35.00 £35.00 -
UK excluding London median hourly rate £35.00 £22.25 -
% change year-on-year +57.30% - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 14,264 16,328 15,535
As % of all contract IT jobs advertised in the UK 15.72% 16.35% 17.06%
Number of daily rates quoted 9,097 10,786 10,049
UK median daily rate £475 £450 £425
Median daily rate % change year-on-year +5.56% +5.88% +3.03%
10th Percentile £275 £263 £258
90th Percentile £638 £625 £600
UK excluding London median daily rate £430 £415 £400
% change year-on-year +3.61% +3.75% +2.56%
Number of hourly rates quoted 316 366 364
UK median hourly rate £20.00 £19.00 £19.00
Median hourly rate % change year-on-year +5.26% - -7.32%
10th Percentile £12.00 £11.55 £11.25
90th Percentile £49.13 £49.38 £52.50
UK excluding London median hourly rate £17.50 £17.50 £18.96
% change year-on-year - -7.70% -7.51%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 20 June 2019.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 20 June 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +13 1,525 £550 - 68
London +31 1,055 £573 +4.09% 42
UK excluding London +4 558 £513 +2.60% 31
South East -29 173 £525 +2.44% 7
North of England +46 162 £500 - 4
Yorkshire +32 93 £500 +5.26% 1
Scotland +79 85 £475 - 5
North West +38 64 £550 - 3
East of England +3 53 £525 +10.53% 7
South West -32 47 £538 +10.26% 4
Midlands -18 26 £500 +5.26% 4
West Midlands +7 18 £500 +5.26% 1
Wales 0 8 £525 -4.55%
East Midlands -13 8 £500 +5.82% 3
North East +16 5 £575 +4.55%
Northern Ireland -5 3 - -

For the 6 months to 20 June 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 1,158 (70.40%) Big Data
2 980 (59.57%) Hadoop
3 957 (58.18%) Python
4 654 (39.76%) Kafka
5 623 (37.87%) SQL
6 583 (35.44%) Apache Hive
7 575 (34.95%) Java
8 573 (34.83%) Agile Software Development
9 559 (33.98%) Amazon AWS
10 539 (32.77%) Scala
11 474 (28.81%) Data Science
12 456 (27.72%) Finance
13 415 (25.23%) Analytics
14 353 (21.46%) Microsoft Azure
15 338 (20.55%) Machine Learning
16 316 (19.21%) NoSQL
17 312 (18.97%) DevOps
18 305 (18.54%) Data Warehouse
19 304 (18.48%) Apache
20 289 (17.57%) Data Modelling
21 281 (17.08%) Google Cloud Platform
22 278 (16.90%) Docker
23 276 (16.78%) ETL
24 268 (16.29%) HBase
25 267 (16.23%) Linux
26 248 (15.08%) Git (software)
27 242 (14.71%) Data Engineering
28 232 (14.10%) Jenkins
29 225 (13.68%) Google
30 202 (12.28%) HDFS

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 304 (18.48%) Apache
2 137 (8.33%) Elasticsearch
3 124 (7.54%) Apache Airflow
4 48 (2.92%) Apache Pig
5 43 (2.61%) Jupyter
6 39 (2.37%) Apache Solr
7 31 (1.88%) SAS
8 27 (1.64%) Luigi
9 24 (1.46%) nginx
10 21 (1.28%) Confluence
11 13 (0.79%) Pachyderm
12 10 (0.61%) Oracle Coherence
13 9 (0.55%) Apache Qpid
14 6 (0.36%) Jetty
14 6 (0.36%) SharePoint
14 6 (0.36%) Tomcat
15 4 (0.24%) ArcGIS
15 4 (0.24%) ColdFusion
15 4 (0.24%) Drupal
16 3 (0.18%) OpenStack
Applications
1 23 (1.40%) Microsoft Excel
2 10 (0.61%) Stata
3 9 (0.55%) SPSS
4 2 (0.12%) Chatbot
4 2 (0.12%) MS Visio
Business Applications
1 14 (0.85%) SAP Oil and Gas
2 4 (0.24%) SAP ERP
2 4 (0.24%) SAP S/4HANA
3 1 (0.061%) Dynamics AX
3 1 (0.061%) Dynamics CRM
3 1 (0.061%) Dynamics NAV
3 1 (0.061%) Oracle Retail
Cloud Services
1 559 (33.98%) Amazon AWS
2 353 (21.46%) Microsoft Azure
3 281 (17.08%) Google Cloud Platform
4 138 (8.39%) Amazon EMR
5 131 (7.96%) Amazon S3
6 130 (7.90%) Serverless
7 97 (5.90%) Amazon Kinesis
8 93 (5.65%) AWS Lambda
9 67 (4.07%) Power Platform
10 60 (3.65%) Amazon EC2
11 58 (3.53%) Azure Data Factory
12 48 (2.92%) GitHub
13 40 (2.43%) PaaS
14 35 (2.13%) Amazon SQS
15 31 (1.88%) Cloud Computing
16 27 (1.64%) AWS CloudFormation
17 21 (1.28%) Azure Machine Learning
18 20 (1.22%) Data Lake Analytics
19 19 (1.16%) CloudFront
20 18 (1.09%) SaaS
Communications & Networking
1 30 (1.82%) Kerberos
2 22 (1.34%) DNS
3 15 (0.91%) HTTP
3 15 (0.91%) TCP/IP
4 14 (0.85%) SSL
5 12 (0.73%) MQTT
6 10 (0.61%) Internet
7 7 (0.43%) Firewall
8 6 (0.36%) FTP
9 5 (0.30%) DSL
10 3 (0.18%) Software-Defined Networking
10 3 (0.18%) SSH
11 2 (0.12%) 2.5G
11 2 (0.12%) 2G
11 2 (0.12%) 3G
11 2 (0.12%) 4G
11 2 (0.12%) 5G
11 2 (0.12%) OpenLDAP
12 1 (0.061%) AMQP
12 1 (0.061%) WAN
Database & Business Intelligence
1 1,158 (70.40%) Big Data
2 980 (59.57%) Hadoop
3 583 (35.44%) Apache Hive
4 316 (19.21%) NoSQL
5 305 (18.54%) Data Warehouse
6 268 (16.29%) HBase
7 188 (11.43%) Impala
8 172 (10.46%) Apache Cassandra
9 168 (10.21%) Data Lake
10 154 (9.36%) MongoDB
11 137 (8.33%) PostgreSQL
12 112 (6.81%) Relational Database
13 106 (6.44%) Amazon Redshift
14 105 (6.38%) SQL Server
15 73 (4.44%) MySQL
16 67 (4.07%) Power BI
17 61 (3.71%) Parquet
18 59 (3.59%) RDBMS
19 56 (3.40%) BigQuery
20 51 (3.10%) Sqoop
Development Applications
1 248 (15.08%) Git (software)
2 232 (14.10%) Jenkins
3 66 (4.01%) JIRA
4 49 (2.98%) Maven
4 49 (2.98%) Subversion
5 30 (1.82%) Atlassian Bamboo
5 30 (1.82%) Visual Studio Team System
6 28 (1.70%) IntelliJ
7 26 (1.58%) Sonatype Nexus
8 25 (1.52%) Gradle
8 25 (1.52%) JUnit
9 24 (1.46%) GitLab
9 24 (1.46%) TeamCity
10 23 (1.40%) Eclipse
11 21 (1.28%) Artifactory
12 19 (1.16%) Mercurial
13 17 (1.03%) RStudio
14 16 (0.97%) Cucumber
15 13 (0.79%) CircleCI
15 13 (0.79%) Travis CI
General
1 456 (27.72%) Finance
2 172 (10.46%) Banking
3 61 (3.71%) Front Office
4 48 (2.92%) Investment Banking
4 48 (2.92%) Retail
5 42 (2.55%) Legal
6 30 (1.82%) Marketing
7 22 (1.34%) Telecoms
8 15 (0.91%) Financial Institution
8 15 (0.91%) Retail Banking
9 11 (0.67%) Advertising
10 8 (0.49%) Manufacturing
11 7 (0.43%) Publishing
12 6 (0.36%) Aerospace
12 6 (0.36%) Automotive
12 6 (0.36%) Pharmaceutical
13 5 (0.30%) General Ledger
14 4 (0.24%) Back Office
14 4 (0.24%) Games
15 2 (0.12%) Social Science
Job Titles
1 622 (37.81%) Data Engineer
2 450 (27.36%) Developer
3 278 (16.90%) Big Data Engineer
4 159 (9.67%) Python Engineer
5 149 (9.06%) Big Data Developer
6 139 (8.45%) Java Developer
7 138 (8.39%) Architect
8 108 (6.57%) Hadoop Engineer
9 107 (6.50%) Data Scientist
10 100 (6.08%) Python Developer
11 99 (6.02%) Data Architect
12 86 (5.23%) Scala Developer
13 83 (5.05%) AWS Engineer
13 83 (5.05%) DevOps Engineer
14 75 (4.56%) Apache Spark Developer
15 67 (4.07%) Hadoop Developer
16 66 (4.01%) Cloud Engineer
17 64 (3.89%) Senior Developer
18 49 (2.98%) Java Engineer
19 46 (2.80%) Senior Data Engineer
Libraries, Frameworks & Software Standards
1 654 (39.76%) Kafka
2 178 (10.82%) PySpark
3 115 (6.99%) Spring
4 85 (5.17%) Spark Streaming
5 84 (5.11%) Pandas
6 73 (4.44%) Natural Language Processing
7 68 (4.13%) Web Services
8 63 (3.83%) NumPy
9 61 (3.71%) Spring Boot
10 57 (3.47%) TensorFlow
11 55 (3.34%) scikit-learn
12 49 (2.98%) REST
13 47 (2.86%) RESTful
14 46 (2.80%) Apache NiFi
15 44 (2.67%) Apache Avro
15 44 (2.67%) RabbitMQ
16 34 (2.07%) Elastic Stack
16 34 (2.07%) JSON
17 31 (1.88%) HTML
17 31 (1.88%) XML
Miscellaneous
1 134 (8.15%) Analytical Skills
2 131 (7.96%) Algorithms
3 110 (6.69%) Greenfield Project
4 73 (4.44%) Data Structures
5 38 (2.31%) Self-Motivation
6 31 (1.88%) Foreign Exchange (FX)
7 24 (1.46%) Market Risk
7 24 (1.46%) Public Cloud
8 22 (1.34%) User Experience
9 20 (1.22%) Cloud Native
9 20 (1.22%) Linux Command Line
10 19 (1.16%) Enterprise Software
11 17 (1.03%) Hedge funds
12 16 (0.97%) Credit Risk
12 16 (0.97%) Grid Computing
13 12 (0.73%) Virtual Team
14 10 (0.61%) Data Protection Act
14 10 (0.61%) Life Science
15 9 (0.55%) Derivative
16 8 (0.49%) N-Tier
Operating Systems
1 267 (16.23%) Linux
2 141 (8.57%) Unix
3 49 (2.98%) Windows
4 14 (0.85%) Red Hat Enterprise Linux
5 6 (0.36%) Android
6 3 (0.18%) Apple iOS
6 3 (0.18%) Solaris
6 3 (0.18%) SUSE Linux Enterprise Server
7 2 (0.12%) CentOS
7 2 (0.12%) Windows Server
8 1 (0.061%) Mesosphere
8 1 (0.061%) Windows Server 2008
8 1 (0.061%) Windows Server 2012
8 1 (0.061%) zOS
Processes & Methodologies
1 573 (34.83%) Agile Software Development
2 474 (28.81%) Data Science
3 415 (25.23%) Analytics
4 338 (20.55%) Machine Learning
5 312 (18.97%) DevOps
6 289 (17.57%) Data Modelling
7 276 (16.78%) ETL
8 242 (14.71%) Data Engineering
9 186 (11.31%) TDD
10 181 (11.00%) Software Engineering
11 158 (9.60%) Continuous Integration
12 152 (9.24%) Artificial Intelligence
13 143 (8.69%) Business Intelligence
14 137 (8.33%) Problem-Solving
15 133 (8.09%) Scrum
16 127 (7.72%) Computer Science
17 124 (7.54%) MapReduce
17 124 (7.54%) Microservices
18 115 (6.99%) Visualisation
19 101 (6.14%) Open Source
Programming Languages
1 957 (58.18%) Python
2 623 (37.87%) SQL
3 575 (34.95%) Java
4 539 (32.77%) Scala
5 179 (10.88%) R
6 78 (4.74%) JavaScript
7 66 (4.01%) C#
8 64 (3.89%) Shell Script
9 60 (3.65%) C++
10 55 (3.34%) Bash Shell
11 43 (2.61%) Java 8
12 40 (2.43%) Ruby
13 36 (2.19%) C
14 28 (1.70%) MATLAB
15 27 (1.64%) PL/SQL
16 25 (1.52%) Go
17 24 (1.46%) T-SQL
18 19 (1.16%) Perl
19 18 (1.09%) U-SQL
20 15 (0.91%) PowerShell
Qualifications
1 103 (6.26%) Degree
2 60 (3.65%) PhD
3 48 (2.92%) Security Cleared
4 38 (2.31%) SC Cleared
5 32 (1.95%) Computer Science Degree
6 28 (1.70%) AWS Certification
7 18 (1.09%) PMI Certification
7 18 (1.09%) PMP
8 16 (0.97%) Master's Degree
9 13 (0.79%) TOGAF Certification
10 12 (0.73%) Numerate Degree
11 5 (0.30%) BPSS Clearance
11 5 (0.30%) DV Cleared
12 4 (0.24%) ISTQB
13 1 (0.061%) CFA
13 1 (0.061%) Postgraduate
Quality Assurance & Compliance
1 74 (4.50%) Data Quality
2 19 (1.16%) GDPR
3 13 (0.79%) QA
4 3 (0.18%) IFRS
4 3 (0.18%) ISO/IEC 27001
5 1 (0.061%) EMIR
5 1 (0.061%) HL7
5 1 (0.061%) MiFID
5 1 (0.061%) PCI DSS
System Software
1 278 (16.90%) Docker
2 202 (12.28%) HDFS
3 87 (5.29%) Apache Flume
4 17 (1.03%) Active Directory
4 17 (1.03%) Apache ZooKeeper
5 4 (0.24%) Virtual Machines
6 3 (0.18%) VMware Infrastructure
7 1 (0.061%) NFS
7 1 (0.061%) Virtual Servers
Systems Management
1 181 (11.00%) Kubernetes
2 122 (7.42%) Ansible
3 101 (6.14%) Oozie
4 97 (5.90%) Hadoop YARN
5 91 (5.53%) Terraform
6 42 (2.55%) Puppet
7 36 (2.19%) Opscode Chef
8 19 (1.16%) Kibana
9 14 (0.85%) Kubeflow
9 14 (0.85%) logstash
10 12 (0.73%) Mesos
11 10 (0.61%) Marathon
12 7 (0.43%) Grafana
13 4 (0.24%) Ambari
13 4 (0.24%) Control-M
13 4 (0.24%) Docker Swarm
13 4 (0.24%) Graphite
13 4 (0.24%) Nagios
13 4 (0.24%) StatsD
14 3 (0.18%) Tivoli
Vendors
1 225 (13.68%) Google
2 152 (9.24%) Cloudera
3 117 (7.11%) Oracle
4 99 (6.02%) Databricks
5 92 (5.59%) Tableau
6 90 (5.47%) Informatica
7 80 (4.86%) Microsoft
8 63 (3.83%) Talend
9 56 (3.40%) Hortonworks
10 44 (2.67%) SAP
11 42 (2.55%) MapR
12 33 (2.01%) Splunk
13 28 (1.70%) IBM
14 19 (1.16%) IBM Sterling
15 18 (1.09%) Teradata
16 15 (0.91%) New Relic
17 13 (0.79%) Atlassian
18 11 (0.67%) MicroStrategy
18 11 (0.67%) Sybase
19 10 (0.61%) Netezza