Period
to 21 October 2017

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 21 October 2017 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
21 Oct 2017
Same period 2016 Same period 2015
Rank 272 358 644
Rank change year-on-year +86 +286 +577
Permanent jobs citing Apache Spark 2203 1603 803
As % of all permanent IT jobs advertised in the UK 1.27% 0.84% 0.37%
As % of the Application Platforms category 7.09% 4.56% 1.92%
Number of salaries quoted 1754 1302 638
UK median annual salary £65,000 £65,000 £65,000
10th Percentile £41,250 £40,000 £40,000
90th Percentile £100,000 £100,000 £91,250
UK excluding London median annual salary £55,000 £55,000 £50,000
% change year-on-year - +10.00% -28.57%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 31075 35135 41716
As % of all permanent IT jobs advertised in the UK 17.85% 18.39% 19.04%
Number of salaries quoted 25033 29672 34749
UK median annual salary £47,500 £45,000 £45,000
Median salary % change year-on-year +5.56% - +5.88%
10th Percentile £27,000 £26,750 £26,250
90th Percentile £76,250 £75,000 £72,500
UK excluding London median annual salary £40,000 £40,000 £40,000

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 21 October 2017.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 16 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 21 October 2017. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +96 2109 £67,000 +1.13% 125
London +80 1507 £70,000 -6.67% 85
UK excluding London +90 671 £55,000 - 42
South East +123 236 £60,000 +4.35% 16
North of England +65 149 £50,000 -12.28% 8
North West +51 116 £52,500 - 7
East of England +112 104 £60,000 +20.00% 4
South West -97 57 £50,000 - 7
Scotland +7 56 £68,750 +37.50% 2
Midlands +1 46 £42,500 -29.17% 5
Yorkshire +18 30 £50,000 -16.67% 1
East Midlands +47 27 £42,500 -5.56% 3
Wales +23 19 £35,000 -36.36%
West Midlands -31 19 £42,500 -29.17% 2
Northern Ireland - 4 £57,000 -
North East +10 3 £35,000 -30.00%

For the 6 months to 21 October 2017, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the number co-occurrences and its proportion to all job ads with a requirement for Apache Spark.

1 1510 (68.54%) Hadoop
2 1492 (67.73%) Big Data
3 1285 (58.33%) Java
4 1224 (55.56%) Python
5 1043 (47.34%) Agile Software Development
6 933 (42.35%) Scala
7 813 (36.90%) Analytics
8 741 (33.64%) SQL
9 683 (31.00%) Machine Learning
10 620 (28.14%) Amazon AWS
11 589 (26.74%) Finance
12 569 (25.83%) Kafka
13 497 (22.56%) Apache Hive
14 466 (21.15%) NoSQL
15 449 (20.38%) Apache Cassandra
16 403 (18.29%) Degree
17 393 (17.84%) Software Engineering
18 381 (17.29%) Apache
19 379 (17.20%) R
20 368 (16.70%) Docker
21 356 (16.16%) TDD
22 349 (15.84%) Scrum
23 345 (15.66%) Business Intelligence
24 344 (15.62%) Continuous Integration
25 338 (15.34%) MongoDB
26 334 (15.16%) Open Source
27 331 (15.02%) Linux
28 324 (14.71%) DevOps
29 320 (14.53%) JavaScript
30 308 (13.98%) Computer Science

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 381 (17.29%) Apache
2 258 (11.71%) Elasticsearch
3 162 (7.35%) Apache Pig
4 132 (5.99%) SAS
5 117 (5.31%) Apache Solr
6 84 (3.81%) Tomcat
7 50 (2.27%) Confluence
8 48 (2.18%) OpenStack
9 43 (1.95%) Apache Airflow
10 41 (1.86%) WebLogic
11 36 (1.63%) JBoss
12 33 (1.50%) WebSphere
13 31 (1.41%) Jetty
14 17 (0.77%) WebSphere Application Server
15 12 (0.54%) CMS
15 12 (0.54%) Oracle SOA Suite
16 10 (0.45%) nginx
17 8 (0.36%) GemFire
18 7 (0.32%) Jupyter
18 7 (0.32%) SharePoint
Applications
1 30 (1.36%) Microsoft Excel
2 23 (1.04%) SPSS
3 8 (0.36%) Microsoft Office
4 6 (0.27%) Spreadsheet
5 4 (0.18%) Microsoft PowerPoint
5 4 (0.18%) S-PLUS
5 4 (0.18%) Stata
6 2 (0.091%) GNU Octave
6 2 (0.091%) Weka
7 1 (0.045%) Chatbot
7 1 (0.045%) HipChat
7 1 (0.045%) LaTeX
Business Applications
1 2 (0.091%) Dynamics CRM
2 1 (0.045%) Capstone
2 1 (0.045%) Markit EDM
Cloud Services
1 620 (28.14%) Amazon AWS
2 266 (12.07%) Microsoft Azure
3 172 (7.81%) Google Cloud Platform
4 85 (3.86%) SaaS
5 79 (3.59%) GitHub
6 67 (3.04%) Amazon EMR
7 55 (2.50%) Amazon Kinesis
8 50 (2.27%) Cloud Computing
9 46 (2.09%) PaaS
10 42 (1.91%) Amazon S3
11 40 (1.82%) Amazon EC2
12 27 (1.23%) AWS Lambda
13 26 (1.18%) Serverless
14 22 (1.00%) IaaS
15 17 (0.77%) Azure Machine Learning
16 15 (0.68%) AWS CloudFormation
16 15 (0.68%) Google Analytics
16 15 (0.68%) Virtual Private Cloud
17 10 (0.45%) Amazon CloudWatch
18 8 (0.36%) OpenShift
Communications & Networking
1 26 (1.18%) Internet
2 19 (0.86%) Kerberos
3 13 (0.59%) HTTP
4 7 (0.32%) DNS
5 6 (0.27%) Network Security
5 6 (0.27%) TCP/IP
6 5 (0.23%) AMQP
6 5 (0.23%) Firewall
6 5 (0.23%) ICMP
6 5 (0.23%) VPN
7 4 (0.18%) LDAP
7 4 (0.18%) SNMP
7 4 (0.18%) Wireshark
8 2 (0.091%) DAB
8 2 (0.091%) LAN
8 2 (0.091%) Software-Defined Networking
8 2 (0.091%) WAN
8 2 (0.091%) Wireless
9 1 (0.045%) FTP
9 1 (0.045%) SSL
Database & Business Intelligence
1 1510 (68.54%) Hadoop
2 1492 (67.73%) Big Data
3 497 (22.56%) Apache Hive
4 466 (21.15%) NoSQL
5 449 (20.38%) Apache Cassandra
6 338 (15.34%) MongoDB
7 285 (12.94%) Relational Database
8 271 (12.30%) HBase
9 242 (10.99%) Data Warehouse
10 212 (9.62%) SQL Server
11 175 (7.94%) PostgreSQL
12 170 (7.72%) MySQL
13 146 (6.63%) Impala
14 143 (6.49%) Amazon Redshift
15 134 (6.08%) RDBMS
16 128 (5.81%) Redis
17 113 (5.13%) Data Mining
18 87 (3.95%) Sqoop
19 84 (3.81%) Neo4j
20 48 (2.18%) Riak
Development Applications
1 228 (10.35%) Git (software)
2 190 (8.62%) Jenkins
3 120 (5.45%) JIRA
4 117 (5.31%) Maven
5 75 (3.40%) Selenium
6 53 (2.41%) Atlassian Bamboo
7 52 (2.36%) JUnit
8 49 (2.22%) Bitbucket
9 42 (1.91%) Eclipse
10 38 (1.72%) Cucumber
11 33 (1.50%) Talend Open Studio
12 30 (1.36%) Gerrit
13 27 (1.23%) Vagrant
14 26 (1.18%) IntelliJ
14 26 (1.18%) Subversion
15 24 (1.09%) Team Foundation Server
16 23 (1.04%) Gradle
16 23 (1.04%) TeamCity
17 21 (0.95%) Mockito
18 20 (0.91%) Jasmine
General
1 589 (26.74%) Finance
2 209 (9.49%) Marketing
3 193 (8.76%) Banking
4 120 (5.45%) Retail
5 118 (5.36%) Telecoms
6 103 (4.68%) Manufacturing
7 101 (4.58%) Electronics
8 81 (3.68%) Investment Banking
9 65 (2.95%) Games
10 63 (2.86%) Front Office
11 56 (2.54%) Legal
12 33 (1.50%) Advertising
12 33 (1.50%) Pharmaceutical
13 32 (1.45%) Retail Banking
14 31 (1.41%) Financial Institution
15 12 (0.54%) Aerospace
16 11 (0.50%) Automotive
17 10 (0.45%) Publishing
18 6 (0.27%) Spanish Language
19 5 (0.23%) Law
Job Titles
1 706 (32.05%) Developer
2 334 (15.16%) Data Engineer
3 333 (15.12%) Java Developer
4 314 (14.25%) Data Scientist
5 208 (9.44%) Architect
6 204 (9.26%) Big Data Engineer
7 150 (6.81%) Scala Developer
8 145 (6.58%) Big Data Developer
9 140 (6.35%) Senior Developer
9 140 (6.35%) Software Engineer
10 94 (4.27%) Hadoop Engineer
11 93 (4.22%) Senior Java Developer
12 92 (4.18%) Data Architect
13 91 (4.13%) Python Developer
14 75 (3.40%) Analyst
15 74 (3.36%) Hadoop Developer
15 74 (3.36%) Java Engineer
16 70 (3.18%) Software Developer
17 69 (3.13%) Java Scala Developer
18 67 (3.04%) Big Data Architect
Libraries, Frameworks & Software Standards
1 569 (25.83%) Kafka
2 211 (9.58%) Spring
2 211 (9.58%) Web Services
3 202 (9.17%) AngularJS
4 158 (7.17%) Node.js
5 137 (6.22%) Natural Language Processing
6 125 (5.67%) CSS
7 120 (5.45%) HTML
8 116 (5.27%) REST
9 115 (5.22%) RabbitMQ
10 112 (5.08%) .NET
11 110 (4.99%) Akka
12 95 (4.31%) React
13 91 (4.13%) RESTful
14 86 (3.90%) Hibernate
15 69 (3.13%) Django
16 68 (3.09%) scikit-learn
17 67 (3.04%) Lucene
18 63 (2.86%) NumPy
19 60 (2.72%) Play Framework
Miscellaneous
1 308 (13.98%) Computer Science
2 284 (12.89%) Analytical Skills
3 268 (12.17%) Algorithms
4 146 (6.63%) Greenfield Project
5 86 (3.90%) Data Structures
6 80 (3.63%) Fintech
7 57 (2.59%) Foreign Exchange (FX)
8 42 (1.91%) Cyberthreat
9 40 (1.82%) Derivative
10 39 (1.77%) Fixed Income
10 39 (1.77%) Public Cloud
11 36 (1.63%) Enterprise Software
12 32 (1.45%) Equities
13 26 (1.18%) Hedge funds
13 26 (1.18%) Mobile App
13 26 (1.18%) User Experience
14 23 (1.04%) Market Risk
15 21 (0.95%) Management Information System
16 18 (0.82%) Clustering
16 18 (0.82%) Credit Risk
Operating Systems
1 331 (15.02%) Linux
2 82 (3.72%) Unix
3 73 (3.31%) Windows
4 23 (1.04%) Android
5 17 (0.77%) Apple iOS
6 14 (0.64%) Red Hat Enterprise Linux
7 13 (0.59%) Ubuntu
8 8 (0.36%) CentOS
9 4 (0.18%) Mac OS X
9 4 (0.18%) openSUSE
9 4 (0.18%) SUSE
10 2 (0.091%) CoreOS
10 2 (0.091%) Solaris
11 1 (0.045%) AIX
11 1 (0.045%) Mesosphere
11 1 (0.045%) Windows Server
Processes & Methodologies
1 1043 (47.34%) Agile Software Development
2 813 (36.90%) Analytics
3 683 (31.00%) Machine Learning
4 393 (17.84%) Software Engineering
5 356 (16.16%) TDD
6 349 (15.84%) Scrum
7 345 (15.66%) Business Intelligence
8 344 (15.62%) Continuous Integration
9 334 (15.16%) Open Source
10 324 (14.71%) DevOps
11 280 (12.71%) Data Analytics
12 259 (11.76%) Visualisation
13 247 (11.21%) MapReduce
14 242 (10.99%) ETL
15 198 (8.99%) Mathematics
16 190 (8.62%) Microservices
17 186 (8.44%) Statistics
18 179 (8.13%) Kanban
19 178 (8.08%) Test Automation
20 177 (8.03%) OO
Programming Languages
1 1285 (58.33%) Java
2 1224 (55.56%) Python
3 933 (42.35%) Scala
4 741 (33.64%) SQL
5 379 (17.20%) R
6 320 (14.53%) JavaScript
7 241 (10.94%) C++
8 210 (9.53%) C#
9 139 (6.31%) Ruby
10 114 (5.17%) C
11 112 (5.08%) Go
12 79 (3.59%) Bash Shell
13 73 (3.31%) MATLAB
14 65 (2.95%) Shell Script
15 64 (2.91%) Perl
16 48 (2.18%) PHP
17 34 (1.54%) Haskell
18 32 (1.45%) Clojure
19 17 (0.77%) ES6
19 17 (0.77%) T-SQL
Qualifications
1 403 (18.29%) Degree
2 137 (6.22%) PhD
3 135 (6.13%) Computer Science Degree
4 96 (4.36%) Security Cleared
5 60 (2.72%) DV Cleared
6 46 (2.09%) Master's Degree
7 41 (1.86%) Numerate Degree
8 33 (1.50%) SC Cleared
9 13 (0.59%) ISEB
10 11 (0.50%) ISTQB
11 10 (0.45%) Postgraduate
12 7 (0.32%) MBA
13 5 (0.23%) AWS Certification
14 3 (0.14%) PMI Certification
15 2 (0.091%) ISTQB Foundation Certification
15 2 (0.091%) Microsoft Certification
16 1 (0.045%) CBAP
16 1 (0.045%) CCA
16 1 (0.045%) CISM
16 1 (0.045%) IIBA
Quality Assurance & Compliance
1 87 (3.95%) Data Quality
2 73 (3.31%) QA
3 8 (0.36%) GDPR
4 6 (0.27%) GCP
5 5 (0.23%) SLA
6 4 (0.18%) Actionable Recommendations
6 4 (0.18%) Sarbanes-Oxley
6 4 (0.18%) Solvency II
7 1 (0.045%) ISO/IEC 27001
7 1 (0.045%) MiFID
7 1 (0.045%) PMO
System Software
1 368 (16.70%) Docker
2 175 (7.94%) HDFS
3 135 (6.13%) Apache Flume
4 31 (1.41%) Apache ZooKeeper
5 11 (0.50%) VMware Infrastructure
6 7 (0.32%) Hyper-V
7 6 (0.27%) Virtual Machines
8 2 (0.091%) VirtualBox
9 1 (0.045%) Active Directory
9 1 (0.045%) Cronacle
9 1 (0.045%) Cygwin
9 1 (0.045%) MapR-FS
Systems Management
1 134 (6.08%) Ansible
2 129 (5.86%) Kubernetes
3 126 (5.72%) Puppet
4 118 (5.36%) Opscode Chef
5 100 (4.54%) Mesos
6 80 (3.63%) Oozie
7 56 (2.54%) Terraform
8 49 (2.22%) Hadoop YARN
9 40 (1.82%) Kibana
10 26 (1.18%) Salt
11 25 (1.13%) logstash
12 16 (0.73%) Marathon
13 15 (0.68%) Grafana
14 13 (0.59%) Ambari
15 9 (0.41%) ITRS Geneos
16 8 (0.36%) Graphite
17 6 (0.27%) Nagios
18 4 (0.18%) Rancher
18 4 (0.18%) ZABBIX
19 2 (0.091%) Docker Swarm
Vendors
1 193 (8.76%) Google
2 158 (7.17%) Oracle
3 157 (7.13%) Tableau
4 148 (6.72%) Cloudera
5 112 (5.08%) SAP
6 104 (4.72%) Microsoft
7 91 (4.13%) Hortonworks
8 48 (2.18%) Splunk
9 42 (1.91%) MapR
10 33 (1.50%) Informatica
10 33 (1.50%) Talend
11 29 (1.32%) Teradata
12 24 (1.09%) TIBCO
13 21 (0.95%) IBM
13 21 (0.95%) RedGate
14 18 (0.82%) MicroStrategy
15 17 (0.77%) VMware
16 14 (0.64%) Red Hat
17 13 (0.59%) Alteryx
17 13 (0.59%) Netezza