Period
to 20 January 2019

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 20 January 2019 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
20 Jan 2019
Same period 2018 Same period 2017
Rank 239 254 326
Rank change year-on-year +15 +72 +213
Permanent jobs citing Apache Spark 2,188 2,312 1,759
As % of all permanent IT jobs advertised in the UK 1.46% 1.34% 0.99%
As % of the Application Platforms category 9.11% 7.82% 5.53%
Number of salaries quoted 1,710 1,928 1,468
UK median annual salary £72,500 £70,000 £67,500
Median salary % change year-on-year +3.57% +3.70% +3.85%
10th Percentile £45,000 £41,925 £40,000
90th Percentile £102,500 £105,000 £98,250
UK excluding London median annual salary £62,500 £52,700 £55,000
% change year-on-year +18.60% -4.18% -4.35%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 24,028 29,569 31,801
As % of all permanent IT jobs advertised in the UK 16.07% 17.08% 17.96%
Number of salaries quoted 19,168 24,326 26,704
UK median annual salary £50,000 £47,500 £45,000
Median salary % change year-on-year +5.26% +5.56% -
10th Percentile £28,000 £27,500 £26,750
90th Percentile £85,000 £77,500 £75,000
UK excluding London median annual salary £43,041 £40,500 £40,000
% change year-on-year +6.27% +1.25% -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 20 January 2019.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 20 January 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +14 2,074 £75,000 +7.14% 316
London -6 1,393 £77,500 +6.90% 209
UK excluding London +93 762 £62,500 +18.60% 128
South East +40 216 £65,000 +18.18% 49
North of England +81 160 £65,000 +23.81% 19
South West +29 129 £57,750 +15.50% 20
Midlands +78 98 £75,000 +76.47% 12
North West +33 97 £70,000 +33.33% 8
Scotland +21 71 £55,000 -12.00% 16
East Midlands +66 70 £75,000 +45.63% 5
East of England +5 69 £67,500 -8.47% 7
Yorkshire +62 49 £50,000 -2.44% 10
West Midlands +46 28 £25,000 -41.18% 7
North East +30 18 £89,995 - 1
Wales +7 12 £62,000 +34.05% 3
Northern Ireland -8 6 £42,500 -50.00% 2
Channel Islands - 1 £37,500 -

For the 6 months to 20 January 2019, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,582 (72.30%) Hadoop
2 1,493 (68.24%) Big Data
3 1,421 (64.95%) Python
4 1,176 (53.75%) Java
5 985 (45.02%) Agile Software Development
6 934 (42.69%) Scala
7 919 (42.00%) Kafka
8 862 (39.40%) Amazon AWS
9 830 (37.93%) SQL
10 795 (36.33%) Analytics
11 752 (34.37%) Machine Learning
12 738 (33.73%) Apache Hive
13 718 (32.82%) NoSQL
14 713 (32.59%) Data Science
15 500 (22.85%) Microsoft Azure
16 495 (22.62%) Software Engineering
17 484 (22.12%) Apache Cassandra
18 474 (21.66%) Apache
19 458 (20.93%) DevOps
20 457 (20.89%) Finance
21 455 (20.80%) Data Modelling
22 446 (20.38%) HBase
23 400 (18.28%) ETL
24 393 (17.96%) Degree
25 383 (17.50%) Artificial Intelligence
26 362 (16.54%) Linux
27 354 (16.18%) Data Engineering
28 340 (15.54%) Google Cloud Platform
29 334 (15.27%) R
30 324 (14.81%) Visualisation

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 474 (21.66%) Apache
2 302 (13.80%) Elasticsearch
3 215 (9.83%) Apache Pig
4 129 (5.90%) SAS
5 104 (4.75%) Apache Solr
6 34 (1.55%) Tomcat
7 32 (1.46%) GlassFish
8 28 (1.28%) Apache Airflow
9 21 (0.96%) Confluence
10 20 (0.91%) JBoss
11 19 (0.87%) WebSphere
12 17 (0.78%) Jupyter
13 16 (0.73%) nginx
14 12 (0.55%) Apache Geronimo
15 11 (0.50%) Cloud Foundry
16 9 (0.41%) Luigi
17 6 (0.27%) Drupal
18 5 (0.23%) CMS
18 5 (0.23%) Sitecore CMS
19 4 (0.18%) GemFire
Applications
1 19 (0.87%) SPSS
2 13 (0.59%) Weka
3 10 (0.46%) Microsoft Excel
4 8 (0.37%) Chatbot
4 8 (0.37%) Stata
5 6 (0.27%) GNU Octave
5 6 (0.27%) Microsoft PowerPoint
6 5 (0.23%) Microsoft Office
6 5 (0.23%) MS Visio
7 3 (0.14%) CPLEX
8 2 (0.091%) S-PLUS
9 1 (0.046%) Microsoft Project
Business Applications
1 1 (0.046%) NetSuite
Cloud Services
1 862 (39.40%) Amazon AWS
2 500 (22.85%) Microsoft Azure
3 340 (15.54%) Google Cloud Platform
4 127 (5.80%) Amazon EMR
5 126 (5.76%) Cloud Computing
6 119 (5.44%) GitHub
7 97 (4.43%) Serverless
8 73 (3.34%) Amazon EC2
8 73 (3.34%) AWS Lambda
9 55 (2.51%) Amazon S3
10 45 (2.06%) Amazon Kinesis
10 45 (2.06%) PaaS
11 37 (1.69%) SaaS
12 35 (1.60%) AWS CloudFormation
13 34 (1.55%) Azure Data Factory
14 26 (1.19%) IaaS
15 25 (1.14%) Cortana Intelligence Suite
16 23 (1.05%) Amazon SageMaker
17 21 (0.96%) Slack
18 19 (0.87%) Amazon SQS
Communications & Networking
1 46 (2.10%) HTTP
2 43 (1.97%) Internet
3 16 (0.73%) AMQP
4 13 (0.59%) DNS
5 12 (0.55%) Kerberos
6 11 (0.50%) Firewall
7 7 (0.32%) Sourcefire
7 7 (0.32%) SSL
8 6 (0.27%) MQTT
9 5 (0.23%) TCP/IP
10 4 (0.18%) DAB
10 4 (0.18%) Network Security
10 4 (0.18%) VPN
11 2 (0.091%) SSH
11 2 (0.091%) WAN
12 1 (0.046%) HTTPS
12 1 (0.046%) IPsec
12 1 (0.046%) LAN
12 1 (0.046%) SAN
12 1 (0.046%) Wi-Fi
Database & Business Intelligence
1 1,582 (72.30%) Hadoop
2 1,493 (68.24%) Big Data
3 738 (33.73%) Apache Hive
4 718 (32.82%) NoSQL
5 484 (22.12%) Apache Cassandra
6 446 (20.38%) HBase
7 324 (14.81%) MongoDB
8 271 (12.39%) Amazon Redshift
9 270 (12.34%) Impala
10 235 (10.74%) Data Warehouse
11 184 (8.41%) SQL Server
12 182 (8.32%) PostgreSQL
13 180 (8.23%) Flink
14 154 (7.04%) Data Lake
15 150 (6.86%) Power BI
16 139 (6.35%) Relational Database
17 138 (6.31%) Sqoop
18 120 (5.48%) MySQL
19 91 (4.16%) CouchDB
20 87 (3.98%) Spark SQL
Development Applications
1 225 (10.28%) Git (software)
2 168 (7.68%) Jenkins
3 108 (4.94%) JIRA
4 59 (2.70%) Visual Studio
5 51 (2.33%) Gradle
6 44 (2.01%) Maven
7 38 (1.74%) Selenium
8 33 (1.51%) Atlassian Bamboo
9 31 (1.42%) GoCD
10 28 (1.28%) JMeter
11 27 (1.23%) Ant
12 26 (1.19%) CircleCI
12 26 (1.19%) JUnit
12 26 (1.19%) SoapUI
12 26 (1.19%) Subversion
13 25 (1.14%) Visual Studio Team System
14 21 (0.96%) Eclipse
14 21 (0.96%) Mercurial
15 20 (0.91%) GitLab
16 19 (0.87%) TeamCity
General
1 457 (20.89%) Finance
2 196 (8.96%) Retail
3 173 (7.91%) Telecoms
4 164 (7.50%) Law
5 156 (7.13%) Banking
6 121 (5.53%) Military
7 86 (3.93%) Marketing
8 62 (2.83%) Games
9 53 (2.42%) Investment Banking
10 46 (2.10%) Manufacturing
11 37 (1.69%) Electronics
12 34 (1.55%) Automotive
13 30 (1.37%) Advertising
14 21 (0.96%) Financial Institution
15 20 (0.91%) Pharmaceutical
16 16 (0.73%) Retail Banking
17 15 (0.69%) Front Office
18 8 (0.37%) Billing
19 7 (0.32%) Legal
20 6 (0.27%) French Language
Job Titles
1 612 (27.97%) Data Engineer
2 499 (22.81%) Developer
3 368 (16.82%) Big Data Engineer
4 292 (13.35%) Data Scientist
5 264 (12.07%) Architect
6 207 (9.46%) Java Developer
7 202 (9.23%) Data Architect
8 122 (5.58%) Senior Data Engineer
9 120 (5.48%) Big Data Developer
9 120 (5.48%) Python Engineer
10 113 (5.16%) Software Engineer
11 109 (4.98%) Hadoop Engineer
12 104 (4.75%) Senior Developer
13 103 (4.71%) Senior Data Scientist
14 95 (4.34%) Consultant
15 90 (4.11%) Big Data Architect
16 76 (3.47%) Java Engineer
17 67 (3.06%) Scala Developer
18 62 (2.83%) Software Developer
19 59 (2.70%) DevOps Engineer
Libraries, Frameworks & Software Standards
1 919 (42.00%) Kafka
2 229 (10.47%) Web Services
3 165 (7.54%) Spark Streaming
4 145 (6.63%) Natural Language Processing
5 137 (6.26%) Node.js
6 136 (6.22%) AngularJS
7 129 (5.90%) HTML
8 124 (5.67%) REST
9 119 (5.44%) Spring
10 110 (5.03%) Pandas
10 110 (5.03%) TensorFlow
11 105 (4.80%) scikit-learn
12 104 (4.75%) RESTful
13 100 (4.57%) React
14 97 (4.43%) HTML5
15 93 (4.25%) .NET
16 92 (4.20%) D3.js
17 86 (3.93%) JSON
18 85 (3.88%) Apache NiFi
19 81 (3.70%) CSS
Miscellaneous
1 263 (12.02%) Algorithms
2 234 (10.69%) Analytical Skills
3 129 (5.90%) Data Structures
4 89 (4.07%) Fintech
5 77 (3.52%) Greenfield Project
6 71 (3.24%) Self-Motivation
7 48 (2.19%) Social Network
8 47 (2.15%) Virtual Team
9 41 (1.87%) Public Cloud
10 38 (1.74%) Management Information System
11 32 (1.46%) Derivative
11 32 (1.46%) User Experience
12 31 (1.42%) Blog
13 30 (1.37%) Life Science
14 29 (1.33%) Social Media
15 25 (1.14%) Linux Command Line
16 21 (0.96%) Internet of Things
17 20 (0.91%) Market Risk
18 19 (0.87%) Video on Demand
19 17 (0.78%) Enterprise Software
Operating Systems
1 362 (16.54%) Linux
2 200 (9.14%) Unix
3 71 (3.24%) Windows
4 23 (1.05%) Android
4 23 (1.05%) Apple iOS
5 10 (0.46%) Red Hat Enterprise Linux
6 9 (0.41%) CentOS
7 8 (0.37%) Ubuntu
8 5 (0.23%) VMS
9 3 (0.14%) Oracle Linux
10 2 (0.091%) DC/OS
10 2 (0.091%) Windows Server
11 1 (0.046%) AIX
11 1 (0.046%) CoreOS
11 1 (0.046%) Mac OS
11 1 (0.046%) Windows Server 2012
Processes & Methodologies
1 985 (45.02%) Agile Software Development
2 795 (36.33%) Analytics
3 752 (34.37%) Machine Learning
4 713 (32.59%) Data Science
5 495 (22.62%) Software Engineering
6 458 (20.93%) DevOps
7 455 (20.80%) Data Modelling
8 400 (18.28%) ETL
9 383 (17.50%) Artificial Intelligence
10 354 (16.18%) Data Engineering
11 324 (14.81%) Visualisation
12 318 (14.53%) Computer Science
13 316 (14.44%) Open Source
14 313 (14.31%) Business Intelligence
15 295 (13.48%) Data Analytics
16 263 (12.02%) TDD
17 244 (11.15%) Mathematics
18 229 (10.47%) Microservices
19 221 (10.10%) Data Analysis
19 221 (10.10%) Scrum
Programming Languages
1 1,421 (64.95%) Python
2 1,176 (53.75%) Java
3 934 (42.69%) Scala
4 830 (37.93%) SQL
5 334 (15.27%) R
6 195 (8.91%) C#
7 190 (8.68%) JavaScript
8 174 (7.95%) C++
9 88 (4.02%) Shell Script
10 81 (3.70%) C
11 76 (3.47%) Java 8
12 69 (3.15%) Bash Shell
13 67 (3.06%) Ruby
14 58 (2.65%) T-SQL
15 52 (2.38%) Perl
16 48 (2.19%) Go
17 44 (2.01%) PHP
18 40 (1.83%) MATLAB
19 37 (1.69%) PL/SQL
20 36 (1.65%) PowerShell
Qualifications
1 393 (17.96%) Degree
2 150 (6.86%) Computer Science Degree
3 82 (3.75%) PhD
4 64 (2.93%) Numerate Degree
5 58 (2.65%) Security Cleared
6 30 (1.37%) SC Cleared
7 25 (1.14%) DV Cleared
8 24 (1.10%) Master's Degree
9 9 (0.41%) CEH
10 7 (0.32%) CISSP
10 7 (0.32%) GCIH
10 7 (0.32%) OSCP
11 4 (0.18%) MCAD
11 4 (0.18%) MCDBA
11 4 (0.18%) MCSD
11 4 (0.18%) Microsoft Certification
11 4 (0.18%) Postgraduate
11 4 (0.18%) TOGAF Certification
12 3 (0.14%) ISEB
12 3 (0.14%) ISTQB
Quality Assurance & Compliance
1 104 (4.75%) Data Quality
2 91 (4.16%) QA
3 31 (1.42%) GDPR
4 11 (0.50%) MiFID
5 8 (0.37%) ISO/IEC 27001
6 7 (0.32%) PCI DSS
7 3 (0.14%) GPG13
7 3 (0.14%) Solvency II
8 1 (0.046%) COBIT
8 1 (0.046%) EMIR
8 1 (0.046%) ISO 9001
8 1 (0.046%) PMO
8 1 (0.046%) SLA
System Software
1 275 (12.57%) Docker
2 244 (11.15%) HDFS
3 172 (7.86%) Apache Flume
4 74 (3.38%) Apache ZooKeeper
5 18 (0.82%) Active Directory
6 17 (0.78%) Virtual Machines
7 14 (0.64%) VirtualBox
8 7 (0.32%) Snort
9 5 (0.23%) VMware Infrastructure
10 2 (0.091%) Hyper-V
11 1 (0.046%) Firmware
11 1 (0.046%) Platform Symphony
11 1 (0.046%) Samba
11 1 (0.046%) VMware ESXi
Systems Management
1 239 (10.92%) Oozie
2 194 (8.87%) Hadoop YARN
3 179 (8.18%) Kubernetes
4 140 (6.40%) Ansible
5 124 (5.67%) Terraform
6 75 (3.43%) Opscode Chef
6 75 (3.43%) Puppet
7 70 (3.20%) Kibana
8 62 (2.83%) Mesos
9 47 (2.15%) Prometheus
10 38 (1.74%) logstash
11 34 (1.55%) Ambari
12 27 (1.23%) Grafana
13 16 (0.73%) Nagios
13 16 (0.73%) Salt
14 14 (0.64%) Docker Swarm
15 7 (0.32%) Oracle ZFS
15 7 (0.32%) Suricata
16 5 (0.23%) ITRS Geneos
16 5 (0.23%) Sysdig
Vendors
1 312 (14.26%) Cloudera
2 278 (12.71%) Tableau
3 254 (11.61%) Hortonworks
4 246 (11.24%) Google
5 212 (9.69%) Microsoft
6 186 (8.50%) Oracle
7 120 (5.48%) Databricks
8 115 (5.26%) SAP
9 108 (4.94%) MapR
10 78 (3.56%) Informatica
11 76 (3.47%) Talend
12 75 (3.43%) IBM
13 50 (2.29%) CA Agile Central
14 48 (2.19%) Splunk
15 36 (1.65%) Alteryx
16 32 (1.46%) Teradata
17 31 (1.42%) TIBCO
18 27 (1.23%) Netezza
19 21 (0.96%) Ab Initio
19 21 (0.96%) Greenplum