Period
to 21 April 2019

The following table provides summary statistics for permanent job vacancies advertised in London with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 21 April 2019 with a comparison to the same period in the previous 2 years.

Apache Spark
England > London
6 months to
21 Apr 2019
Same period 2018 Same period 2017
Rank 133 130 200
Rank change year-on-year -3 +70 +66
Permanent jobs citing Apache Spark 1,536 1,838 1,241
As % of all permanent IT jobs advertised in London 2.53% 2.63% 1.81%
As % of the Application Platforms category 15.36% 14.68% 9.76%
Number of salaries quoted 1,258 1,515 1,066
London median annual salary £80,000 £75,000 £70,000
Median salary % change year-on-year +6.67% +7.14% -
10th Percentile £55,000 £50,000 £49,875
90th Percentile £111,250 £110,000 £107,500
England median annual salary £75,000 £70,000 £67,500
% change year-on-year +7.14% +3.70% +3.85%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies advertised in London with a requirement for application platform skills.

All Application Platform Skills
London
Permanent vacancies with a requirement for application platform skills 9,998 12,517 12,716
As % of all permanent IT jobs advertised in London 16.47% 17.88% 18.53%
Number of salaries quoted 8,089 10,343 10,692
London median annual salary £60,000 £60,000 £55,000
Median salary % change year-on-year - +9.09% +1.85%
10th Percentile £32,950 £34,787 £32,000
90th Percentile £95,000 £90,000 £85,000
England median annual salary £50,000 £50,000 £47,500
% change year-on-year - +5.26% +5.56%

Apache Spark
Job Vacancy Trend in London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in London.

Job vacancy trend for Apache Spark in London

Apache Spark
Salary Trend in London

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark in London.

Salary trend for Apache Spark in London

Apache Spark
Salary Histogram in London

The salary distribution of IT jobs citing Apache Spark in London over the 6 months to 21 April 2019.

Salary histogram for Apache Spark in London

Apache Spark
Job Locations in London

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the London region over the 6 months to 21 April 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -9 241 £80,000 +6.67% 23
West London +25 29 £70,000 +3.70%
East London +1 10 £70,000 -26.32% 3
North London +22 6 £77,500 -
South London -4 5 £71,250 -13.64%
Apache Spark
England

For the 6 months to 21 April 2019, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads across the London region with a requirement for Apache Spark.

1 1,133 (73.76%) Python
2 1,120 (72.92%) Big Data
3 1,078 (70.18%) Hadoop
4 887 (57.75%) Java
5 750 (48.83%) Scala
6 745 (48.50%) Amazon AWS
7 740 (48.18%) Kafka
8 669 (43.55%) Agile Software Development
9 612 (39.84%) Analytics
10 593 (38.61%) SQL
11 591 (38.48%) Machine Learning
12 544 (35.42%) Data Science
13 534 (34.77%) Apache Hive
14 505 (32.88%) NoSQL
15 410 (26.69%) DevOps
16 377 (24.54%) Microsoft Azure
17 369 (24.02%) Software Engineering
18 349 (22.72%) HBase
19 340 (22.14%) Google Cloud Platform
20 327 (21.29%) Apache Cassandra
21 319 (20.77%) Finance
22 315 (20.51%) Cloudera
23 311 (20.25%) ETL
24 304 (19.79%) Artificial Intelligence
25 301 (19.60%) Data Modelling
26 277 (18.03%) Docker
27 258 (16.80%) Linux
28 254 (16.54%) Data Engineering
29 253 (16.47%) Amazon Redshift
30 247 (16.08%) Apache Pig

Apache Spark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 247 (16.08%) Apache Pig
2 245 (15.95%) Elasticsearch
3 242 (15.76%) Apache
4 97 (6.32%) Apache Solr
5 37 (2.41%) SAS
6 29 (1.89%) Apache Airflow
7 14 (0.91%) Jupyter
8 10 (0.65%) Tomcat
9 9 (0.59%) GlassFish
9 9 (0.59%) Luigi
10 7 (0.46%) Confluence
11 6 (0.39%) IPython
12 3 (0.20%) nginx
13 2 (0.13%) ArcGIS
13 2 (0.13%) WebSphere
14 1 (0.065%) Cloud Foundry
Applications
1 10 (0.65%) Microsoft Excel
2 7 (0.46%) SPSS
3 3 (0.20%) Chatbot
3 3 (0.20%) Stata
4 2 (0.13%) GNU Octave
4 2 (0.13%) Microsoft Office
4 2 (0.13%) Microsoft PowerPoint
4 2 (0.13%) MS Visio
Business Applications
1 2 (0.13%) PolarLake
1 2 (0.13%) ReMetrica
1 2 (0.13%) Towers Watson Igloo
2 1 (0.065%) Adobe Campaign
2 1 (0.065%) SAP Oil and Gas
Cloud Services
1 745 (48.50%) Amazon AWS
2 377 (24.54%) Microsoft Azure
3 340 (22.14%) Google Cloud Platform
4 79 (5.14%) Serverless
5 72 (4.69%) Cloud Computing
6 69 (4.49%) Amazon EMR
6 69 (4.49%) Amazon S3
7 64 (4.17%) AWS Lambda
8 52 (3.39%) Amazon EC2
9 51 (3.32%) Amazon Kinesis
10 46 (2.99%) PaaS
11 42 (2.73%) GitHub
12 38 (2.47%) Amazon SQS
13 33 (2.15%) SaaS
14 26 (1.69%) Azure Data Factory
15 17 (1.11%) Amazon ELB
16 16 (1.04%) Route 53
17 15 (0.98%) Amazon SageMaker
18 14 (0.91%) AWS CloudFormation
18 14 (0.91%) Cortana Intelligence Suite
Communications & Networking
1 38 (2.47%) Internet
2 20 (1.30%) HTTP
3 9 (0.59%) LAN
3 9 (0.59%) Software-Defined Networking
3 9 (0.59%) WAN
3 9 (0.59%) Wi-Fi
4 8 (0.52%) AMQP
5 6 (0.39%) MQTT
5 6 (0.39%) TCP/IP
5 6 (0.39%) VPN
6 5 (0.33%) DNS
6 5 (0.33%) Firewall
7 4 (0.26%) FTP
7 4 (0.26%) Kerberos
8 2 (0.13%) SMTP
9 1 (0.065%) HTTPS
9 1 (0.065%) SSL
9 1 (0.065%) Unified Communications
Database & Business Intelligence
1 1,120 (72.92%) Big Data
2 1,078 (70.18%) Hadoop
3 534 (34.77%) Apache Hive
4 505 (32.88%) NoSQL
5 349 (22.72%) HBase
6 327 (21.29%) Apache Cassandra
7 253 (16.47%) Amazon Redshift
8 244 (15.89%) Impala
9 174 (11.33%) MongoDB
10 164 (10.68%) Data Warehouse
11 142 (9.24%) Flink
12 132 (8.59%) PostgreSQL
13 116 (7.55%) Sqoop
14 97 (6.32%) MySQL
15 94 (6.12%) Couchbase
16 93 (6.05%) CouchDB
17 88 (5.73%) Data Lake
18 80 (5.21%) Relational Database
19 79 (5.14%) QlikView
20 77 (5.01%) BigQuery
Development Applications
1 123 (8.01%) Git (software)
2 120 (7.81%) Jenkins
3 42 (2.73%) Maven
4 39 (2.54%) Gradle
5 30 (1.95%) JIRA
6 29 (1.89%) Visual Studio
7 28 (1.82%) Ant
8 22 (1.43%) Atlassian Bamboo
9 17 (1.11%) Selenium
9 17 (1.11%) Travis CI
10 15 (0.98%) Artifactory
11 14 (0.91%) Visual Studio Team System
12 10 (0.65%) Vagrant
13 9 (0.59%) GitLab
13 9 (0.59%) JMeter
14 8 (0.52%) Bitbucket Server
15 7 (0.46%) Subversion
16 6 (0.39%) Cucumber
16 6 (0.39%) JUnit
16 6 (0.39%) TeamCity
General
1 319 (20.77%) Finance
2 234 (15.23%) Law
3 205 (13.35%) Military
4 174 (11.33%) Banking
5 88 (5.73%) Retail
6 59 (3.84%) Telecoms
7 44 (2.86%) Investment Banking
8 39 (2.54%) Marketing
9 32 (2.08%) Games
10 31 (2.02%) Publishing
11 19 (1.24%) Advertising
12 15 (0.98%) Financial Institution
13 12 (0.78%) Automotive
14 10 (0.65%) Front Office
15 9 (0.59%) Billing
16 8 (0.52%) Retail Banking
17 7 (0.46%) Pharmaceutical
18 6 (0.39%) Legal
18 6 (0.39%) Manufacturing
19 3 (0.20%) International Banking
Job Titles
1 603 (39.26%) Data Engineer
2 327 (21.29%) Big Data Engineer
3 243 (15.82%) Developer
4 182 (11.85%) Architect
5 161 (10.48%) Data Architect
6 154 (10.03%) Data Scientist
7 132 (8.59%) Senior Data Engineer
8 128 (8.33%) Python Engineer
9 126 (8.20%) Hadoop Engineer
10 121 (7.88%) Java Developer
11 96 (6.25%) Big Data Architect
12 72 (4.69%) DevOps Engineer
13 71 (4.62%) Big Data Developer
14 64 (4.17%) Java Engineer
14 64 (4.17%) Software Engineer
15 62 (4.04%) Scala Developer
16 49 (3.19%) AWS Engineer
17 45 (2.93%) Solutions Architect
18 43 (2.80%) Consultant
18 43 (2.80%) Python Developer
Libraries, Frameworks & Software Standards
1 740 (48.18%) Kafka
2 136 (8.85%) Web Services
3 107 (6.97%) Apache NiFi
4 102 (6.64%) Natural Language Processing
5 97 (6.32%) Spark Streaming
6 82 (5.34%) REST
7 80 (5.21%) scikit-learn
8 79 (5.14%) Pandas
9 73 (4.75%) RESTful
10 66 (4.30%) NumPy
11 65 (4.23%) .NET
12 64 (4.17%) TensorFlow
13 60 (3.91%) Spring
14 55 (3.58%) Elastic Stack
14 55 (3.58%) Node.js
15 52 (3.39%) PySpark
16 46 (2.99%) Akka
16 46 (2.99%) React
17 39 (2.54%) AngularJS
18 37 (2.41%) Apache Beam
Miscellaneous
1 186 (12.11%) Analytical Skills
2 170 (11.07%) Algorithms
3 135 (8.79%) Data Structures
4 98 (6.38%) Fintech
5 47 (3.06%) Greenfield Project
6 41 (2.67%) Virtual Team
7 27 (1.76%) Management Information System
8 26 (1.69%) Self-Motivation
9 23 (1.50%) Cloud Native
10 22 (1.43%) User Experience
11 20 (1.30%) Mobile App
12 19 (1.24%) Social Media
13 18 (1.17%) Credit Risk
13 18 (1.17%) Life Science
14 14 (0.91%) Linux Command Line
14 14 (0.91%) Public Cloud
15 12 (0.78%) Robotics
16 11 (0.72%) Market Risk
17 10 (0.65%) Analytical Mindset
17 10 (0.65%) Semantic Web
Operating Systems
1 258 (16.80%) Linux
2 111 (7.23%) Unix
3 29 (1.89%) Windows
4 14 (0.91%) CentOS
4 14 (0.91%) Ubuntu
5 10 (0.65%) Android
5 10 (0.65%) Apple iOS
5 10 (0.65%) Windows Server
6 4 (0.26%) Mac OS X
6 4 (0.26%) Oracle Linux
7 2 (0.13%) Debian
8 1 (0.065%) Red Hat Enterprise Linux
Processes & Methodologies
1 669 (43.55%) Agile Software Development
2 612 (39.84%) Analytics
3 591 (38.48%) Machine Learning
4 544 (35.42%) Data Science
5 410 (26.69%) DevOps
6 369 (24.02%) Software Engineering
7 311 (20.25%) ETL
8 304 (19.79%) Artificial Intelligence
9 301 (19.60%) Data Modelling
10 254 (16.54%) Data Engineering
11 230 (14.97%) Data Analytics
12 214 (13.93%) Business Intelligence
13 210 (13.67%) Visualisation
14 203 (13.22%) MapReduce
15 171 (11.13%) Scrum
16 153 (9.96%) Microservices
17 149 (9.70%) Computer Science
18 141 (9.18%) Open Source
19 140 (9.11%) Continuous Integration
20 125 (8.14%) Data Analysis
Programming Languages
1 1,133 (73.76%) Python
2 887 (57.75%) Java
3 750 (48.83%) Scala
4 593 (38.61%) SQL
5 172 (11.20%) R
6 91 (5.92%) C++
7 79 (5.14%) C#
8 75 (4.88%) JavaScript
9 58 (3.78%) Bash Shell
10 51 (3.32%) Java 8
11 41 (2.67%) Shell Script
12 39 (2.54%) Perl
13 35 (2.28%) C
14 34 (2.21%) PL/SQL
15 32 (2.08%) Groovy
16 30 (1.95%) Ruby
17 24 (1.56%) PHP
17 24 (1.56%) T-SQL
18 17 (1.11%) Go
19 10 (0.65%) PowerShell
Qualifications
1 178 (11.59%) Degree
2 66 (4.30%) Computer Science Degree
3 64 (4.17%) PhD
4 40 (2.60%) Numerate Degree
5 28 (1.82%) Security Cleared
6 17 (1.11%) SC Cleared
7 12 (0.78%) Master's Degree
8 11 (0.72%) DV Cleared
9 7 (0.46%) PMI Certification
10 6 (0.39%) TOGAF Certification
11 5 (0.33%) ISTQB
11 5 (0.33%) ISTQB Foundation Certification
11 5 (0.33%) PMP
12 4 (0.26%) MCAD
12 4 (0.26%) MCDBA
12 4 (0.26%) MCSD
12 4 (0.26%) Microsoft Certification
13 2 (0.13%) CEH
14 1 (0.065%) Postgraduate
Quality Assurance & Compliance
1 62 (4.04%) Data Quality
2 27 (1.76%) QA
3 16 (1.04%) GDPR
4 8 (0.52%) MiFID
5 6 (0.39%) PCI DSS
6 3 (0.20%) ISO/IEC 27001
7 1 (0.065%) COBIT
7 1 (0.065%) EMIR
7 1 (0.065%) ISO 15022
7 1 (0.065%) ISO 20022
System Software
1 277 (18.03%) Docker
2 147 (9.57%) Apache Flume
3 90 (5.86%) Apache ZooKeeper
3 90 (5.86%) HDFS
4 8 (0.52%) VirtualBox
5 7 (0.46%) Active Directory
6 1 (0.065%) Virtual Machines
Systems Management
1 231 (15.04%) Oozie
2 157 (10.22%) Hadoop YARN
3 156 (10.16%) Ansible
4 141 (9.18%) Kubernetes
5 102 (6.64%) Terraform
6 88 (5.73%) Opscode Chef
7 69 (4.49%) Puppet
8 48 (3.13%) Ambari
9 40 (2.60%) Mesos
10 30 (1.95%) Kibana
11 29 (1.89%) Nagios
12 24 (1.56%) Prometheus
13 19 (1.24%) Salt
14 15 (0.98%) Marathon
15 13 (0.85%) logstash
16 11 (0.72%) Docker Swarm
17 9 (0.59%) Microsoft Intune
17 9 (0.59%) SCCM
18 4 (0.26%) Control-M
19 3 (0.20%) Consul
Vendors
1 315 (20.51%) Cloudera
2 200 (13.02%) Hortonworks
3 181 (11.78%) Google
4 173 (11.26%) Tableau
5 138 (8.98%) Databricks
6 106 (6.90%) MapR
7 105 (6.84%) Microsoft
8 102 (6.64%) Oracle
9 81 (5.27%) Talend
10 73 (4.75%) Informatica
11 57 (3.71%) IBM
12 24 (1.56%) Splunk
12 24 (1.56%) TIBCO
13 22 (1.43%) SAP
14 20 (1.30%) Netezza
14 20 (1.30%) SaltStack
15 19 (1.24%) Greenplum
16 16 (1.04%) HP
16 16 (1.04%) Sybase
16 16 (1.04%) WorldPay