Period
to 18 February 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 18 February 2018 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
18 Feb 2018
Same period 2017 Same period 2016
Rank 238 308 500
Rank change year-on-year +70 +192 +608
Permanent jobs citing Apache Spark 2,466 1,802 1,199
As % of all permanent IT jobs advertised in the UK 1.42% 1.03% 0.57%
As % of the Application Platforms category 8.36% 5.80% 3.00%
Number of salaries quoted 2,013 1,503 982
UK median annual salary £70,000 £67,500 £65,000
Median salary % change year-on-year +3.70% +3.85% -3.70%
10th Percentile £42,500 £41,250 £41,250
90th Percentile £105,000 £97,500 £95,000
UK excluding London median annual salary £53,100 £55,000 £57,500
% change year-on-year -3.46% -4.35% -8.00%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 29,511 31,082 39,922
As % of all permanent IT jobs advertised in the UK 16.94% 17.80% 18.95%
Number of salaries quoted 24,338 26,011 33,416
UK median annual salary £47,500 £45,000 £45,000
Median salary % change year-on-year +5.56% - +5.88%
10th Percentile £27,500 £26,750 £26,250
90th Percentile £77,500 £75,000 £72,500
UK excluding London median annual salary £41,000 £40,000 £40,000
% change year-on-year +2.50% - +2.56%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 18 February 2018.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 16 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 18 February 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +67 2,350 £70,000 +3.70% 375
London +56 1,734 £75,000 +3.45% 269
UK excluding London +70 703 £53,100 -3.46% 119
South East +105 203 £55,000 -15.38% 30
South West -21 124 £50,000 - 10
North of England -8 122 £52,500 - 36
East of England +82 111 £75,000 +26.57% 13
North West -18 98 £52,500 - 14
Scotland +14 73 £58,750 -9.62% 8
Midlands -43 54 £52,500 -8.70% 17
East Midlands +21 32 £52,500 +16.67% 5
West Midlands -37 22 £56,250 -8.16% 12
Yorkshire +14 20 £52,500 -4.55% 21
Wales +2 13 £52,700 -18.92% 5
North East -35 4 - - 1
Northern Ireland - 3 £85,000 -

For the 6 months to 18 February 2018, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,700 (68.94%) Hadoop
2 1,593 (64.60%) Big Data
3 1,422 (57.66%) Python
4 1,214 (49.23%) Java
5 1,020 (41.36%) Scala
6 984 (39.90%) Agile Software Development
7 959 (38.89%) Analytics
8 887 (35.97%) Machine Learning
9 852 (34.55%) SQL
10 851 (34.51%) Data Science
11 731 (29.64%) Kafka
12 715 (28.99%) Amazon AWS
13 586 (23.76%) Apache Hive
14 561 (22.75%) Finance
15 552 (22.38%) NoSQL
16 484 (19.63%) Apache Cassandra
17 478 (19.38%) Software Engineering
18 442 (17.92%) R
19 434 (17.60%) Linux
20 425 (17.23%) MongoDB
21 421 (17.07%) Degree
22 419 (16.99%) DevOps
23 401 (16.26%) Docker
24 395 (16.02%) Algorithms
25 389 (15.77%) HBase
26 360 (14.60%) Analytical Skills
27 357 (14.48%) ETL
28 352 (14.27%) Computer Science
29 334 (13.54%) Apache
30 326 (13.22%) Visualisation

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 334 (13.54%) Apache
2 311 (12.61%) Elasticsearch
3 151 (6.12%) Apache Pig
4 130 (5.27%) SAS
5 114 (4.62%) Apache Solr
6 40 (1.62%) Apache Airflow
7 31 (1.26%) CMS
8 30 (1.22%) OpenStack
9 28 (1.14%) Tomcat
10 25 (1.01%) Confluence
11 15 (0.61%) nginx
12 14 (0.57%) Luigi
13 13 (0.53%) JBoss
14 12 (0.49%) Jetty
15 11 (0.45%) SharePoint
15 11 (0.45%) WebSphere
16 10 (0.41%) WebLogic
17 9 (0.36%) Oracle SOA Suite
18 8 (0.32%) Jupyter
19 7 (0.28%) IPython
Applications
1 35 (1.42%) Microsoft Excel
2 26 (1.05%) SPSS
3 17 (0.69%) Weka
4 8 (0.32%) Chatbot
5 7 (0.28%) GNU Octave
6 6 (0.24%) Mathematica
7 5 (0.20%) Microsoft PowerPoint
8 2 (0.081%) HipChat
8 2 (0.081%) Microsoft Office
9 1 (0.041%) MS Visio
9 1 (0.041%) S-PLUS
9 1 (0.041%) Stata
Business Applications
1 4 (0.16%) Oracle 11i
1 4 (0.16%) Oracle Assets
1 4 (0.16%) Oracle EBS
1 4 (0.16%) Oracle EBS R12
1 4 (0.16%) Oracle General Ledger
1 4 (0.16%) Oracle iProcurement
1 4 (0.16%) Oracle Payables
1 4 (0.16%) Oracle Receivables
2 3 (0.12%) Dynamics CRM
3 1 (0.041%) Capstone
3 1 (0.041%) Summit
3 1 (0.041%) WebTrends
Cloud Services
1 715 (28.99%) Amazon AWS
2 309 (12.53%) Microsoft Azure
3 127 (5.15%) Google Cloud Platform
4 103 (4.18%) Amazon Kinesis
5 99 (4.01%) GitHub
6 97 (3.93%) Amazon EMR
7 81 (3.28%) Amazon S3
8 73 (2.96%) SaaS
9 44 (1.78%) PaaS
10 40 (1.62%) AWS Lambda
10 40 (1.62%) Cloud Computing
11 37 (1.50%) Serverless
12 33 (1.34%) Amazon EC2
13 30 (1.22%) Azure Machine Learning
14 28 (1.14%) IaaS
15 17 (0.69%) AWS CloudFormation
16 14 (0.57%) Datadog
17 12 (0.49%) Google Analytics
17 12 (0.49%) OpenShift
17 12 (0.49%) Virtual Private Cloud
Communications & Networking
1 36 (1.46%) HTTP
2 34 (1.38%) Internet
3 25 (1.01%) Kerberos
4 21 (0.85%) DNS
5 20 (0.81%) Firewall
6 19 (0.77%) TCP/IP
7 14 (0.57%) VPN
8 9 (0.36%) ICMP
8 9 (0.36%) Network Security
9 8 (0.32%) DAB
10 5 (0.20%) SMTP
11 4 (0.16%) GSM
11 4 (0.16%) LTE
11 4 (0.16%) UMTS
12 3 (0.12%) SSL
13 2 (0.081%) GPRS
13 2 (0.081%) Radio Access Network
13 2 (0.081%) Wireless
14 1 (0.041%) IPsec
14 1 (0.041%) Wi-Fi
Database & Business Intelligence
1 1,700 (68.94%) Hadoop
2 1,593 (64.60%) Big Data
3 586 (23.76%) Apache Hive
4 552 (22.38%) NoSQL
5 484 (19.63%) Apache Cassandra
6 425 (17.23%) MongoDB
7 389 (15.77%) HBase
8 268 (10.87%) Relational Database
9 254 (10.30%) Data Warehouse
10 203 (8.23%) Impala
11 199 (8.07%) PostgreSQL
12 195 (7.91%) SQL Server
13 171 (6.93%) Amazon Redshift
14 162 (6.57%) MySQL
15 149 (6.04%) RDBMS
16 134 (5.43%) Data Mining
17 127 (5.15%) Data Lake
18 99 (4.01%) Power BI
19 98 (3.97%) Flink
20 96 (3.89%) Neo4j
Development Applications
1 209 (8.48%) Git (software)
1 209 (8.48%) Jenkins
2 150 (6.08%) JIRA
3 83 (3.37%) Maven
4 51 (2.07%) Selenium
5 50 (2.03%) Atlassian Bamboo
6 37 (1.50%) Gradle
7 36 (1.46%) Cucumber
8 35 (1.42%) Bitbucket
8 35 (1.42%) JUnit
9 31 (1.26%) Team Foundation Server
10 29 (1.18%) TeamCity
11 27 (1.09%) Sonatype Nexus
12 18 (0.73%) IntelliJ
12 18 (0.73%) Vagrant
13 17 (0.69%) Artifactory
14 15 (0.61%) Travis CI
15 13 (0.53%) Gerrit
15 13 (0.53%) Grunt
15 13 (0.53%) JMeter
General
1 561 (22.75%) Finance
2 161 (6.53%) Banking
3 148 (6.00%) Marketing
4 139 (5.64%) Telecoms
5 128 (5.19%) Retail
6 70 (2.84%) Investment Banking
7 57 (2.31%) Legal
8 56 (2.27%) Electronics
9 47 (1.91%) Front Office
10 46 (1.87%) Manufacturing
11 42 (1.70%) Financial Institution
12 39 (1.58%) Advertising
13 37 (1.50%) Games
14 22 (0.89%) Pharmaceutical
15 21 (0.85%) Retail Banking
16 19 (0.77%) Law
17 18 (0.73%) Automotive
18 13 (0.53%) Publishing
19 7 (0.28%) Aerospace
20 4 (0.16%) Corporate Banking
Job Titles
1 597 (24.21%) Developer
2 515 (20.88%) Data Engineer
3 368 (14.92%) Data Scientist
4 295 (11.96%) Big Data Engineer
5 253 (10.26%) Architect
6 248 (10.06%) Java Developer
7 160 (6.49%) Data Architect
7 160 (6.49%) Software Engineer
8 140 (5.68%) Big Data Developer
9 126 (5.11%) Scala Developer
10 116 (4.70%) Java Engineer
11 107 (4.34%) Senior Developer
12 106 (4.30%) Senior Data Scientist
13 100 (4.06%) Big Data Architect
14 99 (4.01%) Hadoop Engineer
15 93 (3.77%) DevOps Engineer
15 93 (3.77%) Machine Learning Engineer
16 83 (3.37%) Python Developer
17 81 (3.28%) AWS Engineer
18 70 (2.84%) Senior Data Engineer
Libraries, Frameworks & Software Standards
1 731 (29.64%) Kafka
2 208 (8.43%) Natural Language Processing
3 195 (7.91%) Node.js
4 186 (7.54%) Spring
5 174 (7.06%) HTML
6 166 (6.73%) AngularJS
7 161 (6.53%) Web Services
8 151 (6.12%) CSS
9 129 (5.23%) REST
10 110 (4.46%) .NET
10 110 (4.46%) React
11 104 (4.22%) Akka
12 103 (4.18%) Pandas
12 103 (4.18%) Spark Streaming
13 95 (3.85%) RabbitMQ
14 93 (3.77%) HTML5
14 93 (3.77%) NumPy
15 90 (3.65%) Hibernate
16 76 (3.08%) Django
17 71 (2.88%) TensorFlow
Miscellaneous
1 395 (16.02%) Algorithms
2 360 (14.60%) Analytical Skills
3 352 (14.27%) Computer Science
4 139 (5.64%) Fintech
5 120 (4.87%) Greenfield Project
6 94 (3.81%) Data Structures
7 68 (2.76%) User Experience
8 59 (2.39%) Cyberthreat
9 47 (1.91%) Clustering
10 44 (1.78%) Hedge funds
11 41 (1.66%) Data Centre
12 33 (1.34%) Mobile App
13 30 (1.22%) Fixed Income
14 27 (1.09%) Enterprise Software
14 27 (1.09%) Equities
15 25 (1.01%) Derivative
16 24 (0.97%) Xbox
17 23 (0.93%) Linux Command Line
18 22 (0.89%) Market Risk
18 22 (0.89%) Social Media
Operating Systems
1 434 (17.60%) Linux
2 124 (5.03%) Unix
3 68 (2.76%) Windows
4 17 (0.69%) Ubuntu
5 11 (0.45%) Red Hat Enterprise Linux
6 10 (0.41%) Android
7 9 (0.36%) Apple iOS
8 7 (0.28%) CentOS
9 5 (0.20%) CoreOS
9 5 (0.20%) Mac OS X
10 4 (0.16%) Windows Server
11 2 (0.081%) Mac OS
11 2 (0.081%) Mesosphere
11 2 (0.081%) Solaris
12 1 (0.041%) AIX
12 1 (0.041%) Amazon Linux
12 1 (0.041%) DC/OS
12 1 (0.041%) VMS
Processes & Methodologies
1 984 (39.90%) Agile Software Development
2 959 (38.89%) Analytics
3 887 (35.97%) Machine Learning
4 851 (34.51%) Data Science
5 478 (19.38%) Software Engineering
6 419 (16.99%) DevOps
7 357 (14.48%) ETL
8 326 (13.22%) Visualisation
9 315 (12.77%) Business Intelligence
10 313 (12.69%) Data Analytics
11 310 (12.57%) TDD
12 296 (12.00%) Continuous Integration
13 291 (11.80%) Open Source
14 276 (11.19%) Scrum
15 268 (10.87%) Mathematics
16 256 (10.38%) Microservices
17 227 (9.21%) Statistics
18 211 (8.56%) Data Modelling
19 210 (8.52%) MapReduce
20 207 (8.39%) Data Visualisation
Programming Languages
1 1,422 (57.66%) Python
2 1,214 (49.23%) Java
3 1,020 (41.36%) Scala
4 852 (34.55%) SQL
5 442 (17.92%) R
6 317 (12.85%) JavaScript
7 260 (10.54%) C#
8 178 (7.22%) C++
9 165 (6.69%) Ruby
10 117 (4.74%) Go
11 110 (4.46%) Bash Shell
12 98 (3.97%) C
13 93 (3.77%) MATLAB
14 71 (2.88%) Shell Script
15 55 (2.23%) Perl
16 28 (1.14%) PL/SQL
17 25 (1.01%) PHP
17 25 (1.01%) T-SQL
18 23 (0.93%) Haskell
18 23 (0.93%) PowerShell
Qualifications
1 421 (17.07%) Degree
2 174 (7.06%) PhD
3 140 (5.68%) Computer Science Degree
4 93 (3.77%) Security Cleared
5 75 (3.04%) Master's Degree
6 59 (2.39%) DV Cleared
7 32 (1.30%) Numerate Degree
8 23 (0.93%) ISTQB
9 16 (0.65%) SC Cleared
10 13 (0.53%) CISSP
11 12 (0.49%) CEH
11 12 (0.49%) GCIH
11 12 (0.49%) ISTQB Foundation Certification
11 12 (0.49%) OSCP
12 11 (0.45%) ISEB
12 11 (0.45%) MBA
13 7 (0.28%) AWS Certification
14 4 (0.16%) PRINCE2 Certification
15 3 (0.12%) Microsoft Certification
16 2 (0.081%) Postgraduate
Quality Assurance & Compliance
1 102 (4.14%) Data Quality
2 92 (3.73%) QA
3 18 (0.73%) GCP
4 14 (0.57%) ISO/IEC 27001
5 12 (0.49%) GPG13
5 12 (0.49%) Solvency II
6 10 (0.41%) GDPR
7 7 (0.28%) SLA
8 5 (0.20%) Sarbanes-Oxley
9 4 (0.16%) Actionable Recommendations
9 4 (0.16%) MiFID
10 3 (0.12%) EMIR
11 2 (0.081%) Dodd-Frank
11 2 (0.081%) FATCA
11 2 (0.081%) PMO
12 1 (0.041%) HIPAA
System Software
1 401 (16.26%) Docker
2 209 (8.48%) HDFS
3 158 (6.41%) Apache Flume
4 46 (1.87%) Apache ZooKeeper
5 23 (0.93%) VMware Infrastructure
6 18 (0.73%) Virtual Machines
7 15 (0.61%) Hyper-V
8 12 (0.49%) Snort
9 5 (0.20%) VirtualBox
10 4 (0.16%) LXC
10 4 (0.16%) Xen
11 3 (0.12%) KVM
12 2 (0.081%) Active Directory
12 2 (0.081%) MapR-FS
13 1 (0.041%) GridGain
Systems Management
1 168 (6.81%) Kubernetes
2 158 (6.41%) Ansible
3 140 (5.68%) Hadoop YARN
4 126 (5.11%) Puppet
5 94 (3.81%) Terraform
6 88 (3.57%) Oozie
7 79 (3.20%) Mesos
8 73 (2.96%) Opscode Chef
9 30 (1.22%) Kibana
10 19 (0.77%) Ambari
10 19 (0.77%) Salt
11 15 (0.61%) logstash
12 12 (0.49%) Nagios
12 12 (0.49%) Suricata
13 9 (0.36%) Docker Swarm
13 9 (0.36%) Marathon
14 8 (0.32%) ITRS Geneos
15 7 (0.28%) Prometheus
15 7 (0.28%) ZABBIX
16 6 (0.24%) Rancher
Vendors
1 248 (10.06%) Cloudera
2 225 (9.12%) Google
3 221 (8.96%) Tableau
4 202 (8.19%) Oracle
5 182 (7.38%) Microsoft
6 131 (5.31%) Hortonworks
7 82 (3.33%) Talend
8 67 (2.72%) Informatica
8 67 (2.72%) SAP
9 60 (2.43%) Teradata
10 57 (2.31%) MapR
11 54 (2.19%) IBM
12 50 (2.03%) Alteryx
13 44 (1.78%) VMware
14 36 (1.46%) Twitter
15 33 (1.34%) Splunk
16 32 (1.30%) Atlassian
17 27 (1.09%) TIBCO
18 25 (1.01%) Greenplum
19 21 (0.85%) RedGate