Period
to 24 August 2017

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 24 August 2017 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
24 Aug 2017
Same period 2016 Same period 2015
Rank 273 378 727
Rank change year-on-year +105 +349 +505
Permanent jobs citing Apache Spark 2152 1613 644
As % of all permanent IT jobs advertised in the UK 1.22% 0.82% 0.30%
As % of the Application Platforms category 6.82% 4.42% 1.54%
Number of salaries quoted 1705 1314 534
UK median annual salary £67,500 £65,000 £65,000
Median salary % change year-on-year +3.85% - -7.14%
10th Percentile £42,100 £40,000 £41,250
90th Percentile £100,000 £105,000 £93,750
UK excluding London median annual salary £60,000 £55,000 £52,500
% change year-on-year +9.09% +4.76% -25.00%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 31567 36495 41760
As % of all permanent IT jobs advertised in the UK 17.96% 18.58% 19.28%
Number of salaries quoted 25390 31031 34555
UK median annual salary £47,500 £45,000 £45,000
Median salary % change year-on-year +5.56% - +5.88%
10th Percentile £27,000 £26,250 £26,250
90th Percentile £76,250 £75,000 £72,500
UK excluding London median annual salary £41,000 £40,000 £40,000
% change year-on-year +2.50% - +3.90%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 24 August 2017.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 16 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 24 August 2017. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +99 2060 £67,500 +3.85% 172
London +70 1388 £70,000 -3.45% 137
UK excluding London +123 742 £60,000 +9.09% 37
South East +156 236 £65,000 +18.18% 15
North of England +110 191 £55,000 -8.33% 9
North West +89 133 £60,000 +41.18% 6
East of England +108 129 £50,000 -16.67% 5
Scotland +7 55 £72,500 +45.00% 2
Midlands +17 54 £42,500 -29.17% 4
South West -79 50 £52,500 +5.00% 2
Yorkshire +24 47 £52,500 -12.50% 2
East Midlands +45 36 £42,500 -34.62%
Wales +17 23 £35,000 -36.36%
West Midlands -18 18 £60,000 +4.35% 4
North East +18 11 £35,000 -48.15% 1
Northern Ireland - 4 £57,000 -

For the 6 months to 24 August 2017, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the number co-occurrences and its proportion to all job ads with a requirement for Apache Spark.

1 1529 (71.05%) Big Data
2 1472 (68.40%) Hadoop
3 1317 (61.20%) Java
4 1171 (54.41%) Python
5 991 (46.05%) Agile Software Development
6 875 (40.66%) Scala
7 727 (33.78%) Analytics
8 680 (31.60%) SQL
9 626 (29.09%) Machine Learning
10 574 (26.67%) Amazon AWS
11 551 (25.60%) Kafka
12 539 (25.05%) Apache Hive
13 508 (23.61%) NoSQL
14 501 (23.28%) Apache Cassandra
15 493 (22.91%) Finance
16 431 (20.03%) Degree
17 384 (17.84%) R
18 380 (17.66%) Software Engineering
19 378 (17.57%) Apache
20 377 (17.52%) TDD
21 370 (17.19%) MongoDB
22 364 (16.91%) Scrum
23 351 (16.31%) Open Source
24 327 (15.20%) Business Intelligence
25 325 (15.10%) JavaScript
26 320 (14.87%) Linux
27 319 (14.82%) Docker
28 300 (13.94%) Computer Science
29 296 (13.75%) Elasticsearch
30 292 (13.57%) HBase

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 378 (17.57%) Apache
2 296 (13.75%) Elasticsearch
3 191 (8.88%) Apache Pig
4 157 (7.30%) Apache Solr
5 130 (6.04%) SAS
6 117 (5.44%) Tomcat
7 58 (2.70%) OpenStack
8 55 (2.56%) JBoss
9 53 (2.46%) WebLogic
10 49 (2.28%) Confluence
11 41 (1.91%) WebSphere
12 40 (1.86%) Jetty
13 38 (1.77%) Apache Airflow
14 21 (0.98%) WebSphere Application Server
15 17 (0.79%) GlassFish
16 12 (0.56%) GemFire
16 12 (0.56%) Oracle SOA Suite
17 11 (0.51%) Jupyter
18 9 (0.42%) nginx
19 8 (0.37%) CMS
Applications
1 28 (1.30%) SPSS
2 19 (0.88%) Microsoft Excel
3 7 (0.33%) Microsoft Office
4 6 (0.28%) Spreadsheet
5 5 (0.23%) S-PLUS
5 5 (0.23%) Stata
6 3 (0.14%) Microsoft PowerPoint
6 3 (0.14%) SPSS Modeler
6 3 (0.14%) Weka
7 1 (0.046%) GNU Octave
7 1 (0.046%) InVision
7 1 (0.046%) LaTeX
Business Applications
1 1 (0.046%) Markit EDM
Cloud Services
1 574 (26.67%) Amazon AWS
2 248 (11.52%) Microsoft Azure
3 169 (7.85%) Google Cloud Platform
4 94 (4.37%) GitHub
5 65 (3.02%) Amazon EMR
6 62 (2.88%) SaaS
7 49 (2.28%) Amazon Kinesis
7 49 (2.28%) Cloud Computing
8 44 (2.04%) Amazon EC2
9 42 (1.95%) Amazon S3
10 31 (1.44%) PaaS
11 19 (0.88%) AWS Lambda
12 16 (0.74%) IaaS
13 14 (0.65%) Google Analytics
14 11 (0.51%) OpenShift
15 10 (0.46%) Azure Data Factory
16 9 (0.42%) Virtual Private Cloud
17 8 (0.37%) Azure Machine Learning
18 7 (0.33%) Amazon CloudWatch
19 6 (0.28%) Bluemix
Communications & Networking
1 32 (1.49%) Internet
2 20 (0.93%) Kerberos
3 11 (0.51%) LDAP
4 10 (0.46%) HTTP
5 6 (0.28%) DNS
6 5 (0.23%) AMQP
6 5 (0.23%) LAN
6 5 (0.23%) Network Security
6 5 (0.23%) WAN
6 5 (0.23%) Wireless
6 5 (0.23%) Wireshark
7 4 (0.19%) SNMP
8 3 (0.14%) Software-Defined Networking
9 2 (0.093%) FTP
10 1 (0.046%) Intranet
10 1 (0.046%) MQTT
10 1 (0.046%) SSL
10 1 (0.046%) TCP/IP
10 1 (0.046%) tcpdump
Database & Business Intelligence
1 1529 (71.05%) Big Data
2 1472 (68.40%) Hadoop
3 539 (25.05%) Apache Hive
4 508 (23.61%) NoSQL
5 501 (23.28%) Apache Cassandra
6 370 (17.19%) MongoDB
7 292 (13.57%) HBase
8 271 (12.59%) Relational Database
9 202 (9.39%) Data Warehouse
9 202 (9.39%) SQL Server
10 197 (9.15%) MySQL
11 146 (6.78%) PostgreSQL
11 146 (6.78%) Redis
12 141 (6.55%) Impala
13 129 (5.99%) Amazon Redshift
14 124 (5.76%) Data Mining
15 121 (5.62%) RDBMS
16 73 (3.39%) Neo4j
17 70 (3.25%) Sqoop
18 64 (2.97%) Riak
Development Applications
1 209 (9.71%) Git (software)
2 168 (7.81%) Jenkins
3 112 (5.20%) Maven
4 91 (4.23%) JIRA
5 66 (3.07%) Selenium
6 56 (2.60%) JUnit
7 47 (2.18%) Eclipse
8 41 (1.91%) Talend Open Studio
9 35 (1.63%) Bitbucket
10 34 (1.58%) Atlassian Bamboo
11 33 (1.53%) Cucumber
11 33 (1.53%) Mockito
12 32 (1.49%) Subversion
13 30 (1.39%) TeamCity
14 29 (1.35%) Vagrant
15 25 (1.16%) Gerrit
16 23 (1.07%) Hudson
17 21 (0.98%) Gradle
18 20 (0.93%) IntelliJ
19 19 (0.88%) JBehave
General
1 493 (22.91%) Finance
2 223 (10.36%) Marketing
3 173 (8.04%) Banking
4 130 (6.04%) Telecoms
5 117 (5.44%) Manufacturing
6 116 (5.39%) Electronics
6 116 (5.39%) Retail
7 77 (3.58%) Games
8 68 (3.16%) Investment Banking
9 55 (2.56%) Front Office
10 35 (1.63%) Pharmaceutical
11 34 (1.58%) Advertising
12 32 (1.49%) Legal
13 23 (1.07%) Retail Banking
14 14 (0.65%) Financial Institution
15 12 (0.56%) Publishing
16 10 (0.46%) Aerospace
17 6 (0.28%) Automotive
18 5 (0.23%) Spanish Language
19 4 (0.19%) Law
Job Titles
1 714 (33.18%) Developer
2 362 (16.82%) Java Developer
3 352 (16.36%) Data Engineer
4 309 (14.36%) Data Scientist
5 231 (10.73%) Architect
6 211 (9.80%) Big Data Engineer
7 158 (7.34%) Software Engineer
8 155 (7.20%) Senior Developer
9 154 (7.16%) Scala Developer
10 144 (6.69%) Big Data Developer
11 120 (5.58%) Hadoop Engineer
12 105 (4.88%) Java Engineer
13 103 (4.79%) Data Architect
14 95 (4.41%) Senior Java Developer
15 90 (4.18%) Apache Spark Developer
16 86 (4.00%) Python Developer
17 85 (3.95%) Big Data Architect
18 83 (3.86%) Software Developer
19 80 (3.72%) Hadoop Developer
20 76 (3.53%) Cloud Architect
Libraries, Frameworks & Software Standards
1 551 (25.60%) Kafka
2 228 (10.59%) AngularJS
3 221 (10.27%) Spring
4 217 (10.08%) Web Services
5 156 (7.25%) Node.js
6 137 (6.37%) .NET
6 137 (6.37%) REST
7 111 (5.16%) Lucene
7 111 (5.16%) RabbitMQ
8 110 (5.11%) HTML
9 104 (4.83%) React
10 97 (4.51%) CSS
11 94 (4.37%) Natural Language Processing
12 92 (4.28%) Hibernate
13 86 (4.00%) Akka
14 82 (3.81%) RESTful
15 71 (3.30%) Elastic Stack
16 67 (3.11%) XML
17 61 (2.83%) JMS
18 60 (2.79%) scikit-learn
Miscellaneous
1 300 (13.94%) Computer Science
2 264 (12.27%) Analytical Skills
3 257 (11.94%) Algorithms
4 137 (6.37%) Greenfield Project
5 81 (3.76%) Data Structures
6 65 (3.02%) Foreign Exchange (FX)
7 60 (2.79%) Fintech
8 55 (2.56%) Enterprise Software
9 41 (1.91%) Public Cloud
10 37 (1.72%) Derivative
11 29 (1.35%) Fixed Income
12 26 (1.21%) Semantic Web
13 23 (1.07%) Mobile App
14 21 (0.98%) Cyberthreat
14 21 (0.98%) Management Information System
15 20 (0.93%) User Experience
16 19 (0.88%) Credit Risk
16 19 (0.88%) Life Science
16 19 (0.88%) Market Risk
17 15 (0.70%) PKI
Operating Systems
1 320 (14.87%) Linux
2 77 (3.58%) Unix
2 77 (3.58%) Windows
3 30 (1.39%) Android
4 25 (1.16%) Apple iOS
5 16 (0.74%) Red Hat Enterprise Linux
6 9 (0.42%) VMS
7 8 (0.37%) CentOS
8 4 (0.19%) Mac OS X
8 4 (0.19%) openSUSE
8 4 (0.19%) SUSE
8 4 (0.19%) Ubuntu
9 3 (0.14%) CoreOS
10 2 (0.093%) Windows Server
11 1 (0.046%) Mesosphere
Processes & Methodologies
1 991 (46.05%) Agile Software Development
2 727 (33.78%) Analytics
3 626 (29.09%) Machine Learning
4 380 (17.66%) Software Engineering
5 377 (17.52%) TDD
6 364 (16.91%) Scrum
7 351 (16.31%) Open Source
8 327 (15.20%) Business Intelligence
9 288 (13.38%) Continuous Integration
10 287 (13.34%) DevOps
11 253 (11.76%) Visualisation
12 245 (11.38%) MapReduce
13 230 (10.69%) Data Analytics
14 229 (10.64%) ETL
15 196 (9.11%) Kanban
16 184 (8.55%) OO
17 183 (8.50%) Mathematics
17 183 (8.50%) Statistics
18 172 (7.99%) Test Automation
19 171 (7.95%) Mentoring
Programming Languages
1 1317 (61.20%) Java
2 1171 (54.41%) Python
3 875 (40.66%) Scala
4 680 (31.60%) SQL
5 384 (17.84%) R
6 325 (15.10%) JavaScript
7 262 (12.17%) C++
8 237 (11.01%) C#
9 118 (5.48%) C
10 103 (4.79%) Go
11 101 (4.69%) Ruby
12 86 (4.00%) Perl
13 77 (3.58%) MATLAB
14 75 (3.49%) Bash Shell
15 74 (3.44%) Shell Script
16 63 (2.93%) PHP
17 31 (1.44%) Haskell
18 29 (1.35%) Clojure
19 28 (1.30%) T-SQL
20 25 (1.16%) PL/SQL
Qualifications
1 431 (20.03%) Degree
2 144 (6.69%) PhD
3 119 (5.53%) Computer Science Degree
4 85 (3.95%) Security Cleared
5 51 (2.37%) Master's Degree
6 48 (2.23%) DV Cleared
6 48 (2.23%) Numerate Degree
7 32 (1.49%) SC Cleared
8 18 (0.84%) Postgraduate
9 5 (0.23%) ISTQB
10 4 (0.19%) ISEB
11 3 (0.14%) PMI Certification
12 2 (0.093%) CCA
12 2 (0.093%) CISM
12 2 (0.093%) CISSP
13 1 (0.046%) Citrix Certification
13 1 (0.046%) MBA
13 1 (0.046%) MCSA
13 1 (0.046%) MCSD
13 1 (0.046%) Microsoft Certification
Quality Assurance & Compliance
1 75 (3.49%) QA
2 70 (3.25%) Data Quality
3 13 (0.60%) GCP
4 11 (0.51%) Sarbanes-Oxley
5 5 (0.23%) Actionable Recommendations
6 4 (0.19%) GDPR
7 1 (0.046%) MiFID
7 1 (0.046%) PMO
System Software
1 319 (14.82%) Docker
2 171 (7.95%) HDFS
3 116 (5.39%) Apache Flume
4 28 (1.30%) Apache ZooKeeper
5 10 (0.46%) VMware Infrastructure
6 9 (0.42%) LXC
6 9 (0.42%) Virtual Machines
7 7 (0.33%) Hyper-V
8 3 (0.14%) VirtualBox
9 2 (0.093%) Cronacle
9 2 (0.093%) KVM
10 1 (0.046%) Active Directory
10 1 (0.046%) Cygwin
10 1 (0.046%) Xen
Systems Management
1 161 (7.48%) Opscode Chef
2 154 (7.16%) Ansible
3 151 (7.02%) Puppet
4 106 (4.93%) Mesos
5 91 (4.23%) Kubernetes
6 77 (3.58%) Oozie
7 74 (3.44%) Kibana
8 43 (2.00%) logstash
9 37 (1.72%) Terraform
10 35 (1.63%) Hadoop YARN
11 33 (1.53%) Salt
12 14 (0.65%) Grafana
13 10 (0.46%) Ambari
13 10 (0.46%) Marathon
14 9 (0.42%) Graphite
15 7 (0.33%) Nagios
15 7 (0.33%) ZABBIX
16 5 (0.23%) SCCM
17 4 (0.19%) Capistrano
18 3 (0.14%) Microsoft Intune
Vendors
1 188 (8.74%) Google
2 154 (7.16%) Oracle
3 145 (6.74%) Tableau
4 137 (6.37%) Cloudera
5 123 (5.72%) SAP
6 78 (3.62%) Hortonworks
7 74 (3.44%) Microsoft
8 48 (2.23%) MapR
9 43 (2.00%) Splunk
10 41 (1.91%) Talend
11 36 (1.67%) Informatica
12 25 (1.16%) Teradata
13 23 (1.07%) TIBCO
14 19 (0.88%) IBM
15 18 (0.84%) Alias
15 18 (0.84%) Red Hat
16 17 (0.79%) Apigee
16 17 (0.79%) Apple
17 16 (0.74%) MicroStrategy
18 14 (0.65%) Netezza