Period
to 13 December 2017

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 13 December 2017 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
13 Dec 2017
Same period 2016 Same period 2015
Rank 255 336 576
Rank change year-on-year +81 +240 +608
Permanent jobs citing Apache Spark 2397 1742 962
As % of all permanent IT jobs advertised in the UK 1.33% 0.94% 0.45%
As % of the Application Platforms category 7.71% 5.15% 2.40%
Number of salaries quoted 1960 1445 771
UK median annual salary £67,500 £67,500 £65,000
Median salary % change year-on-year - +3.85% -7.14%
10th Percentile £42,250 £40,400 £41,250
90th Percentile £102,500 £100,000 £92,500
UK excluding London median annual salary £52,500 £55,000 £55,000
% change year-on-year -4.55% - -21.43%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 31107 33803 40112
As % of all permanent IT jobs advertised in the UK 17.27% 18.18% 18.74%
Number of salaries quoted 25260 28479 33486
UK median annual salary £47,500 £45,000 £45,000
Median salary % change year-on-year +5.56% - +5.88%
10th Percentile £27,000 £26,750 £26,250
90th Percentile £77,500 £75,000 £73,750
UK excluding London median annual salary £40,000 £40,000 £40,000

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 13 December 2017.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 13 December 2017. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +77 2307 £67,500 - 177
London +59 1700 £70,000 -6.67% 139
UK excluding London +81 674 £52,500 -4.55% 41
South East +118 227 £57,500 -1.71% 7
North of England +28 128 £52,500 - 15
East of England +111 104 £60,000 +11.63% 11
North West +10 101 £52,500 -4.55% 13
South West -56 99 £50,000 -4.76% 4
Scotland +3 50 £65,000 +30.00% 2
Midlands -12 41 £42,500 -29.17% 1
Yorkshire +25 27 £52,500 -10.64% 1
East Midlands +29 24 £42,500 -5.56% 1
Wales +17 22 £38,187 -36.35%
West Midlands -23 17 £42,500 -34.62%
Northern Ireland - 3 £85,000 - 1

For the 6 months to 13 December 2017, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the number co-occurrences and its proportion to all job ads with a requirement for Apache Spark.

1 1667 (69.55%) Hadoop
2 1585 (66.12%) Big Data
3 1382 (57.66%) Python
4 1281 (53.44%) Java
5 1051 (43.85%) Agile Software Development
6 976 (40.72%) Scala
7 915 (38.17%) Analytics
8 834 (34.79%) Machine Learning
9 807 (33.67%) SQL
10 700 (29.20%) Amazon AWS
11 641 (26.74%) Kafka
12 609 (25.41%) Finance
13 562 (23.45%) Apache Hive
14 516 (21.53%) NoSQL
15 435 (18.15%) Software Engineering
16 431 (17.98%) Apache Cassandra
17 422 (17.61%) Docker
18 421 (17.56%) Degree
19 408 (17.02%) DevOps
20 403 (16.81%) Linux
21 401 (16.73%) R
22 397 (16.56%) MongoDB
23 360 (15.02%) Apache
24 340 (14.18%) Business Intelligence
25 335 (13.98%) Algorithms
26 334 (13.93%) JavaScript
27 333 (13.89%) Computer Science
28 321 (13.39%) Scrum
29 319 (13.31%) Analytical Skills
30 313 (13.06%) Continuous Integration

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 360 (15.02%) Apache
2 290 (12.10%) Elasticsearch
3 162 (6.76%) Apache Pig
4 135 (5.63%) SAS
5 99 (4.13%) Apache Solr
6 54 (2.25%) Apache Airflow
7 51 (2.13%) Tomcat
8 45 (1.88%) Confluence
9 43 (1.79%) OpenStack
10 24 (1.00%) WebSphere
11 21 (0.88%) JBoss
12 19 (0.79%) Jetty
13 18 (0.75%) WebLogic
14 15 (0.63%) CMS
15 13 (0.54%) SharePoint
16 11 (0.46%) nginx
17 10 (0.42%) Jupyter
18 8 (0.33%) GemFire
18 8 (0.33%) Luigi
18 8 (0.33%) Oracle SOA Suite
Applications
1 38 (1.59%) Microsoft Excel
2 31 (1.29%) SPSS
3 7 (0.29%) Microsoft Office
4 6 (0.25%) Microsoft PowerPoint
5 5 (0.21%) Spreadsheet
6 4 (0.17%) Mathematica
7 3 (0.13%) Chatbot
7 3 (0.13%) GNU Octave
7 3 (0.13%) S-PLUS
7 3 (0.13%) Stata
7 3 (0.13%) Weka
8 2 (0.083%) HipChat
9 1 (0.042%) LaTeX
Business Applications
1 3 (0.13%) Dynamics CRM
2 1 (0.042%) Capstone
2 1 (0.042%) Summit
Cloud Services
1 700 (29.20%) Amazon AWS
2 299 (12.47%) Microsoft Azure
3 161 (6.72%) Google Cloud Platform
4 97 (4.05%) SaaS
5 95 (3.96%) GitHub
6 81 (3.38%) Amazon Kinesis
7 74 (3.09%) Amazon EMR
8 59 (2.46%) PaaS
9 57 (2.38%) Amazon S3
10 45 (1.88%) Cloud Computing
11 36 (1.50%) AWS Lambda
12 32 (1.34%) Amazon EC2
13 30 (1.25%) Serverless
14 26 (1.08%) IaaS
15 25 (1.04%) Azure Machine Learning
16 20 (0.83%) Virtual Private Cloud
17 16 (0.67%) AWS CloudFormation
18 12 (0.50%) Amazon CloudWatch
18 12 (0.50%) Google Analytics
19 10 (0.42%) Azure Data Factory
Communications & Networking
1 31 (1.29%) Kerberos
2 28 (1.17%) Internet
3 19 (0.79%) HTTP
4 14 (0.58%) DNS
5 13 (0.54%) Firewall
6 12 (0.50%) TCP/IP
7 10 (0.42%) VPN
8 9 (0.38%) ICMP
9 8 (0.33%) LDAP
9 8 (0.33%) Network Security
10 6 (0.25%) DAB
11 3 (0.13%) SSL
12 2 (0.083%) GSM
12 2 (0.083%) LTE
12 2 (0.083%) UMTS
13 1 (0.042%) LAN
13 1 (0.042%) MQTT
13 1 (0.042%) Skype
13 1 (0.042%) Software-Defined Networking
13 1 (0.042%) Wi-Fi
Database & Business Intelligence
1 1667 (69.55%) Hadoop
2 1585 (66.12%) Big Data
3 562 (23.45%) Apache Hive
4 516 (21.53%) NoSQL
5 431 (17.98%) Apache Cassandra
6 397 (16.56%) MongoDB
7 309 (12.89%) HBase
8 295 (12.31%) Relational Database
9 252 (10.51%) Data Warehouse
10 223 (9.30%) SQL Server
11 202 (8.43%) PostgreSQL
12 174 (7.26%) MySQL
13 168 (7.01%) Amazon Redshift
14 157 (6.55%) Impala
15 151 (6.30%) RDBMS
16 123 (5.13%) Data Mining
17 107 (4.46%) Redis
18 99 (4.13%) Sqoop
19 86 (3.59%) Neo4j
20 74 (3.09%) BigQuery
Development Applications
1 236 (9.85%) Git (software)
2 194 (8.09%) Jenkins
3 160 (6.68%) JIRA
4 92 (3.84%) Maven
5 57 (2.38%) Bitbucket
6 55 (2.29%) Atlassian Bamboo
6 55 (2.29%) Selenium
7 49 (2.04%) JUnit
8 44 (1.84%) Cucumber
9 38 (1.59%) Eclipse
10 34 (1.42%) IntelliJ
11 32 (1.34%) Talend Open Studio
11 32 (1.34%) Team Foundation Server
12 28 (1.17%) TeamCity
13 27 (1.13%) Vagrant
14 24 (1.00%) Gradle
15 20 (0.83%) Mockito
16 19 (0.79%) Gerrit
16 19 (0.79%) Sonatype Nexus
17 18 (0.75%) Subversion
General
1 609 (25.41%) Finance
2 205 (8.55%) Marketing
3 181 (7.55%) Banking
4 150 (6.26%) Telecoms
5 142 (5.92%) Retail
6 103 (4.30%) Electronics
7 98 (4.09%) Manufacturing
8 75 (3.13%) Legal
9 70 (2.92%) Investment Banking
10 61 (2.54%) Front Office
11 49 (2.04%) Games
12 40 (1.67%) Advertising
12 40 (1.67%) Financial Institution
13 33 (1.38%) Pharmaceutical
14 27 (1.13%) Retail Banking
15 16 (0.67%) Law
16 14 (0.58%) Automotive
17 8 (0.33%) Aerospace
18 7 (0.29%) Publishing
19 5 (0.21%) Corporate Banking
Job Titles
1 675 (28.16%) Developer
2 412 (17.19%) Data Engineer
3 349 (14.56%) Data Scientist
4 290 (12.10%) Java Developer
5 246 (10.26%) Big Data Engineer
6 210 (8.76%) Architect
7 160 (6.68%) Software Engineer
8 138 (5.76%) Big Data Developer
9 137 (5.72%) Scala Developer
10 135 (5.63%) Senior Developer
11 116 (4.84%) Data Architect
12 105 (4.38%) Python Developer
13 104 (4.34%) Hadoop Engineer
14 95 (3.96%) Java Engineer
15 90 (3.75%) Senior Java Developer
16 83 (3.46%) DevOps Engineer
17 77 (3.21%) Big Data Architect
18 75 (3.13%) Hadoop Developer
19 72 (3.00%) Analyst
20 69 (2.88%) Java Scala Developer
Libraries, Frameworks & Software Standards
1 641 (26.74%) Kafka
2 223 (9.30%) Spring
3 201 (8.39%) Node.js
4 191 (7.97%) Natural Language Processing
5 190 (7.93%) Web Services
6 179 (7.47%) AngularJS
7 164 (6.84%) CSS
8 161 (6.72%) HTML
9 136 (5.67%) REST
10 120 (5.01%) RabbitMQ
11 107 (4.46%) React
12 106 (4.42%) Akka
13 89 (3.71%) .NET
14 86 (3.59%) Django
15 83 (3.46%) Hibernate
16 79 (3.30%) HTML5
17 78 (3.25%) scikit-learn
18 75 (3.13%) Pandas
18 75 (3.13%) RESTful
19 71 (2.96%) Elastic Stack
Miscellaneous
1 335 (13.98%) Algorithms
2 333 (13.89%) Computer Science
3 319 (13.31%) Analytical Skills
4 141 (5.88%) Greenfield Project
5 115 (4.80%) Fintech
6 84 (3.50%) Data Structures
7 58 (2.42%) Cyberthreat
8 44 (1.84%) User Experience
9 41 (1.71%) Clustering
10 38 (1.59%) Derivative
10 38 (1.59%) Fixed Income
11 37 (1.54%) Hedge funds
12 32 (1.34%) Mobile App
12 32 (1.34%) Public Cloud
13 31 (1.29%) Enterprise Software
14 30 (1.25%) Data Centre
15 28 (1.17%) Equities
15 28 (1.17%) Market Risk
16 23 (0.96%) Foreign Exchange (FX)
17 22 (0.92%) Credit Risk
Operating Systems
1 403 (16.81%) Linux
2 109 (4.55%) Unix
3 70 (2.92%) Windows
4 21 (0.88%) Android
5 15 (0.63%) Apple iOS
5 15 (0.63%) Ubuntu
6 14 (0.58%) Red Hat Enterprise Linux
7 5 (0.21%) CentOS
8 4 (0.17%) Mac OS X
8 4 (0.17%) openSUSE
8 4 (0.17%) SUSE
9 3 (0.13%) CoreOS
10 2 (0.083%) Mesosphere
10 2 (0.083%) Solaris
11 1 (0.042%) AIX
11 1 (0.042%) Windows Server
Processes & Methodologies
1 1051 (43.85%) Agile Software Development
2 915 (38.17%) Analytics
3 834 (34.79%) Machine Learning
4 435 (18.15%) Software Engineering
5 408 (17.02%) DevOps
6 340 (14.18%) Business Intelligence
7 321 (13.39%) Scrum
8 313 (13.06%) Continuous Integration
9 312 (13.02%) TDD
10 308 (12.85%) Data Analytics
11 301 (12.56%) Open Source
12 297 (12.39%) ETL
13 288 (12.02%) Visualisation
14 237 (9.89%) Microservices
15 234 (9.76%) Mathematics
16 231 (9.64%) MapReduce
17 216 (9.01%) Statistics
18 195 (8.14%) Artificial Intelligence
19 194 (8.09%) Test Automation
20 172 (7.18%) BDD
Programming Languages
1 1382 (57.66%) Python
2 1281 (53.44%) Java
3 976 (40.72%) Scala
4 807 (33.67%) SQL
5 401 (16.73%) R
6 334 (13.93%) JavaScript
7 223 (9.30%) C#
8 203 (8.47%) C++
9 182 (7.59%) Ruby
10 134 (5.59%) Go
11 112 (4.67%) C
12 106 (4.42%) Bash Shell
13 78 (3.25%) Shell Script
14 73 (3.05%) MATLAB
15 51 (2.13%) Perl
16 33 (1.38%) PHP
17 24 (1.00%) Haskell
18 23 (0.96%) ES6
18 23 (0.96%) PL/SQL
19 22 (0.92%) Clojure
Qualifications
1 421 (17.56%) Degree
2 159 (6.63%) PhD
3 144 (6.01%) Computer Science Degree
4 104 (4.34%) Security Cleared
5 67 (2.80%) DV Cleared
5 67 (2.80%) Master's Degree
6 41 (1.71%) Numerate Degree
7 32 (1.34%) SC Cleared
8 15 (0.63%) ISEB
8 15 (0.63%) ISTQB
9 12 (0.50%) CISSP
10 11 (0.46%) CEH
10 11 (0.46%) GCIH
10 11 (0.46%) OSCP
11 9 (0.38%) MBA
12 6 (0.25%) AWS Certification
12 6 (0.25%) Postgraduate
13 4 (0.17%) ISTQB Foundation Certification
14 3 (0.13%) PMI Certification
15 2 (0.083%) Microsoft Certification
Quality Assurance & Compliance
1 86 (3.59%) Data Quality
2 77 (3.21%) QA
3 13 (0.54%) GCP
3 13 (0.54%) ISO/IEC 27001
4 11 (0.46%) GPG13
5 8 (0.33%) GDPR
6 7 (0.29%) Actionable Recommendations
6 7 (0.29%) SLA
7 5 (0.21%) Solvency II
8 4 (0.17%) Sarbanes-Oxley
9 1 (0.042%) MiFID
9 1 (0.042%) PMO
System Software
1 422 (17.61%) Docker
2 194 (8.09%) HDFS
3 137 (5.72%) Apache Flume
4 38 (1.59%) Apache ZooKeeper
5 17 (0.71%) Virtual Machines
5 17 (0.71%) VMware Infrastructure
6 14 (0.58%) Hyper-V
7 11 (0.46%) Snort
8 3 (0.13%) VirtualBox
9 2 (0.083%) MapR-FS
10 1 (0.042%) Active Directory
10 1 (0.042%) Cronacle
10 1 (0.042%) Cygwin
10 1 (0.042%) GridGain
Systems Management
1 175 (7.30%) Kubernetes
2 162 (6.76%) Ansible
3 117 (4.88%) Puppet
4 103 (4.30%) Hadoop YARN
5 101 (4.21%) Mesos
6 97 (4.05%) Opscode Chef
7 94 (3.92%) Terraform
8 86 (3.59%) Oozie
9 31 (1.29%) Kibana
10 21 (0.88%) Salt
11 18 (0.75%) logstash
12 15 (0.63%) Ambari
12 15 (0.63%) Marathon
13 11 (0.46%) Grafana
13 11 (0.46%) Suricata
14 9 (0.38%) Graphite
14 9 (0.38%) ITRS Geneos
15 7 (0.29%) Nagios
16 6 (0.25%) QRadar
17 5 (0.21%) Rancher
Vendors
1 213 (8.89%) Google
2 194 (8.09%) Oracle
3 179 (7.47%) Cloudera
4 172 (7.18%) Tableau
5 142 (5.92%) Microsoft
6 109 (4.55%) SAP
7 101 (4.21%) Hortonworks
8 54 (2.25%) Teradata
9 50 (2.09%) Splunk
10 42 (1.75%) MapR
11 36 (1.50%) VMware
12 35 (1.46%) Informatica
13 32 (1.34%) Talend
14 31 (1.29%) Twitter
15 28 (1.17%) RedGate
16 26 (1.08%) IBM
17 25 (1.04%) Atlassian
18 22 (0.92%) Alteryx
19 18 (0.75%) TIBCO
20 16 (0.67%) Netezza