Period
to 27 June 2019

The following table provides summary statistics for contract job vacancies advertised in England with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 27 June 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK > England
6 months to
27 Jun 2019
Same period 2018 Same period 2017
Rank 152 163 214
Rank change year-on-year +11 +51 +184
Contract jobs citing Apache Spark 1,521 1,506 1,054
As % of all contract IT jobs advertised in England 1.83% 1.63% 1.24%
As % of the Application Platforms category 11.63% 9.93% 7.26%
Number of daily rates quoted 1,122 1,085 749
England median daily rate £550 £550 £550
10th Percentile £450 £421 £400
90th Percentile £710 £688 £688
UK median daily rate £550 £550 £550
Number of hourly rates quoted 1 7 0
England median hourly rate £35.00 £22.50 -
Median hourly rate % change year-on-year +55.56% - -
UK median hourly rate £35.00 £35.00 -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in England with a requirement for application platform skills.

All Application Platform Skills
England
Contract vacancies with a requirement for application platform skills 13,076 15,161 14,513
As % of all contract IT jobs advertised in England 15.71% 16.43% 17.11%
Number of daily rates quoted 8,399 10,124 9,415
England median daily rate £475 £450 £438
Median daily rate % change year-on-year +5.56% +2.86% +2.94%
10th Percentile £275 £268 £263
90th Percentile £641 £625 £606
UK median daily rate £475 £450 £425
% change year-on-year +5.56% +5.88% +3.03%
Number of hourly rates quoted 287 347 345
England median hourly rate £20.00 £20.00 £21.00
Median hourly rate % change year-on-year - -4.76% +5.00%
10th Percentile £12.00 £11.66 £11.25
90th Percentile £50.00 £51.25 £52.50
UK median hourly rate £20.00 £19.75 £19.63
% change year-on-year +1.27% +0.64% -4.27%

Apache Spark
Job Vacancy Trend in England

Job postings citing Apache Spark as a percentage of all IT jobs advertised in England.

Job vacancy trend for Apache Spark in England

Apache Spark
Contractor Daily Rate Trend in England

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in England.

Contractor daily rate trend for Apache Spark in England

Apache Spark
Contractor Daily Rate Histogram in England

The daily rate distribution of IT jobs citing Apache Spark in England over the 6 months to 27 June 2019.

Contractor daily rate histogram for Apache Spark in England

Apache Spark
Contractor Hourly Rate Trend in England

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark in England.

Contractor hourly rate trend for Apache Spark in England

Apache Spark
Contract Job Locations in England

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the England region over the 6 months to 27 June 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London +30 1,060 £575 +4.55% 51
South East -37 167 £525 +2.44% 8
North of England +61 161 £500 - 4
Yorkshire +32 91 £500 +5.26% 1
North West +45 67 £550 - 3
East of England +9 52 £522 +9.89% 6
South West -13 46 £538 +10.26% 6
Midlands +1 26 £513 +7.89% 2
West Midlands +11 18 £500 +5.26%
East Midlands -8 8 £519 +7.51% 2
North East +23 3 £575 +4.55%
South Coast - 1 - -
Apache Spark
UK

For the 6 months to 27 June 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the England region with a requirement for Apache Spark.

1 1,062 (69.82%) Big Data
2 916 (60.22%) Hadoop
3 902 (59.30%) Python
4 595 (39.12%) Kafka
5 584 (38.40%) SQL
6 545 (35.83%) Agile Software Development
7 538 (35.37%) Amazon AWS
8 520 (34.19%) Java
9 516 (33.93%) Apache Hive
10 493 (32.41%) Scala
11 447 (29.39%) Data Science
12 412 (27.09%) Finance
13 385 (25.31%) Analytics
14 330 (21.70%) Microsoft Azure
15 304 (19.99%) NoSQL
16 301 (19.79%) Apache
17 296 (19.46%) Machine Learning
18 291 (19.13%) DevOps
19 289 (19.00%) Data Warehouse
20 279 (18.34%) Google Cloud Platform
21 275 (18.08%) Docker
22 264 (17.36%) Data Modelling
23 263 (17.29%) Linux
24 255 (16.77%) ETL
25 238 (15.65%) HBase
26 237 (15.58%) Data Engineering
27 228 (14.99%) Google
28 226 (14.86%) Git (software)
29 222 (14.60%) Jenkins
30 184 (12.10%) TDD

Apache Spark
Co-occurring IT Skills in England by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 301 (19.79%) Apache
2 132 (8.68%) Elasticsearch
3 123 (8.09%) Apache Airflow
4 46 (3.02%) Apache Pig
5 40 (2.63%) Jupyter
6 38 (2.50%) Apache Solr
7 29 (1.91%) SAS
8 27 (1.78%) Luigi
9 24 (1.58%) nginx
10 21 (1.38%) Confluence
11 13 (0.85%) Pachyderm
12 10 (0.66%) Oracle Coherence
13 9 (0.59%) Apache Qpid
14 6 (0.39%) Jetty
14 6 (0.39%) SharePoint
14 6 (0.39%) Tomcat
15 4 (0.26%) ColdFusion
15 4 (0.26%) Drupal
16 3 (0.20%) ArcGIS
16 3 (0.20%) Cloud Foundry
Applications
1 22 (1.45%) Microsoft Excel
2 10 (0.66%) Stata
3 9 (0.59%) SPSS
4 2 (0.13%) Chatbot
5 1 (0.066%) MS Visio
Business Applications
1 14 (0.92%) SAP Oil and Gas
2 4 (0.26%) SAP ERP
2 4 (0.26%) SAP S/4HANA
3 1 (0.066%) Cognos Business Intelligence
3 1 (0.066%) Dynamics AX
3 1 (0.066%) Dynamics CRM
3 1 (0.066%) Dynamics NAV
3 1 (0.066%) Oracle Retail
Cloud Services
1 538 (35.37%) Amazon AWS
2 330 (21.70%) Microsoft Azure
3 279 (18.34%) Google Cloud Platform
4 138 (9.07%) Amazon EMR
5 129 (8.48%) Serverless
6 128 (8.42%) Amazon S3
7 95 (6.25%) Amazon Kinesis
8 92 (6.05%) AWS Lambda
9 64 (4.21%) Power Platform
10 60 (3.94%) Amazon EC2
11 57 (3.75%) Azure Data Factory
12 49 (3.22%) GitHub
13 37 (2.43%) PaaS
14 35 (2.30%) Amazon SQS
15 27 (1.78%) AWS CloudFormation
16 21 (1.38%) Cloud Computing
17 20 (1.31%) Data Lake Analytics
18 19 (1.25%) Azure Machine Learning
18 19 (1.25%) CloudFront
19 18 (1.18%) Amazon SageMaker
Communications & Networking
1 29 (1.91%) Kerberos
2 22 (1.45%) DNS
3 14 (0.92%) HTTP
3 14 (0.92%) SSL
3 14 (0.92%) TCP/IP
4 12 (0.79%) MQTT
5 9 (0.59%) Internet
6 6 (0.39%) Firewall
6 6 (0.39%) FTP
7 5 (0.33%) DSL
8 3 (0.20%) Software-Defined Networking
9 2 (0.13%) 2.5G
9 2 (0.13%) 2G
9 2 (0.13%) 3G
9 2 (0.13%) 4G
9 2 (0.13%) 5G
9 2 (0.13%) OpenLDAP
10 1 (0.066%) LAN
10 1 (0.066%) SSH
10 1 (0.066%) WAN
Database & Business Intelligence
1 1,062 (69.82%) Big Data
2 916 (60.22%) Hadoop
3 516 (33.93%) Apache Hive
4 304 (19.99%) NoSQL
5 289 (19.00%) Data Warehouse
6 238 (15.65%) HBase
7 168 (11.05%) Data Lake
8 160 (10.52%) Apache Cassandra
9 145 (9.53%) Impala
10 143 (9.40%) MongoDB
11 133 (8.74%) PostgreSQL
12 108 (7.10%) Amazon Redshift
13 100 (6.57%) SQL Server
14 99 (6.51%) Relational Database
15 69 (4.54%) MySQL
16 64 (4.21%) Power BI
17 56 (3.68%) Sqoop
18 53 (3.48%) BigQuery
18 53 (3.48%) Parquet
19 50 (3.29%) Amazon Athena
Development Applications
1 226 (14.86%) Git (software)
2 222 (14.60%) Jenkins
3 68 (4.47%) JIRA
4 38 (2.50%) Maven
5 37 (2.43%) Subversion
6 31 (2.04%) Atlassian Bamboo
7 29 (1.91%) Visual Studio Team System
8 26 (1.71%) Gradle
8 26 (1.71%) Sonatype Nexus
9 24 (1.58%) GitLab
9 24 (1.58%) TeamCity
10 21 (1.38%) Artifactory
11 19 (1.25%) IntelliJ
11 19 (1.25%) Mercurial
12 17 (1.12%) RStudio
13 16 (1.05%) Cucumber
13 16 (1.05%) JUnit
14 14 (0.92%) Travis CI
15 13 (0.85%) CircleCI
15 13 (0.85%) Selenium
General
1 412 (27.09%) Finance
2 152 (9.99%) Banking
3 60 (3.94%) Front Office
4 48 (3.16%) Retail
5 42 (2.76%) Investment Banking
6 40 (2.63%) Legal
7 29 (1.91%) Marketing
8 20 (1.31%) Telecoms
9 15 (0.99%) Financial Institution
10 12 (0.79%) Advertising
11 11 (0.72%) Retail Banking
12 9 (0.59%) Publishing
13 7 (0.46%) Manufacturing
14 6 (0.39%) Aerospace
14 6 (0.39%) Automotive
15 5 (0.33%) Games
15 5 (0.33%) General Ledger
16 4 (0.26%) Back Office
17 3 (0.20%) Pharmaceutical
18 2 (0.13%) Social Science
Job Titles
1 608 (39.97%) Data Engineer
2 391 (25.71%) Developer
3 272 (17.88%) Big Data Engineer
4 160 (10.52%) Python Engineer
5 122 (8.02%) Architect
5 122 (8.02%) Java Developer
6 119 (7.82%) Big Data Developer
7 106 (6.97%) Hadoop Engineer
8 93 (6.11%) Python Developer
9 87 (5.72%) Data Architect
9 87 (5.72%) Data Scientist
10 84 (5.52%) AWS Engineer
11 78 (5.13%) DevOps Engineer
12 74 (4.87%) Apache Spark Developer
13 73 (4.80%) Scala Developer
14 65 (4.27%) Cloud Engineer
15 64 (4.21%) Senior Developer
16 59 (3.88%) Hadoop Developer
17 47 (3.09%) Java Engineer
18 44 (2.89%) Administrator
Libraries, Frameworks & Software Standards
1 595 (39.12%) Kafka
2 176 (11.57%) PySpark
3 91 (5.98%) Spring
4 85 (5.59%) Spark Streaming
5 80 (5.26%) Pandas
6 56 (3.68%) Natural Language Processing
7 53 (3.48%) TensorFlow
8 52 (3.42%) Web Services
9 50 (3.29%) NumPy
10 49 (3.22%) RESTful
11 46 (3.02%) Apache NiFi
11 46 (3.02%) Spring Boot
12 45 (2.96%) RabbitMQ
13 44 (2.89%) scikit-learn
14 43 (2.83%) REST
15 34 (2.24%) Apache Avro
16 32 (2.10%) JSON
17 31 (2.04%) HTML
18 30 (1.97%) Elastic Stack
18 30 (1.97%) XML
Miscellaneous
1 120 (7.89%) Algorithms
2 109 (7.17%) Analytical Skills
3 108 (7.10%) Greenfield Project
4 65 (4.27%) Data Structures
5 40 (2.63%) Self-Motivation
6 31 (2.04%) Foreign Exchange (FX)
7 24 (1.58%) Market Risk
7 24 (1.58%) Public Cloud
8 21 (1.38%) Linux Command Line
9 20 (1.31%) User Experience
10 18 (1.18%) Enterprise Software
11 17 (1.12%) Cloud Native
11 17 (1.12%) Credit Risk
11 17 (1.12%) Hedge funds
12 16 (1.05%) Grid Computing
13 12 (0.79%) Virtual Team
14 11 (0.72%) Data Protection Act
15 10 (0.66%) Life Science
16 9 (0.59%) Derivative
17 6 (0.39%) Replication
Operating Systems
1 263 (17.29%) Linux
2 118 (7.76%) Unix
3 48 (3.16%) Windows
4 14 (0.92%) Red Hat Enterprise Linux
5 5 (0.33%) Android
6 3 (0.20%) Apple iOS
6 3 (0.20%) SUSE Linux Enterprise Server
7 2 (0.13%) CentOS
7 2 (0.13%) Solaris
7 2 (0.13%) Windows Server
8 1 (0.066%) Mesosphere
8 1 (0.066%) Windows Server 2008
8 1 (0.066%) Windows Server 2012
8 1 (0.066%) zOS
Processes & Methodologies
1 545 (35.83%) Agile Software Development
2 447 (29.39%) Data Science
3 385 (25.31%) Analytics
4 296 (19.46%) Machine Learning
5 291 (19.13%) DevOps
6 264 (17.36%) Data Modelling
7 255 (16.77%) ETL
8 237 (15.58%) Data Engineering
9 184 (12.10%) TDD
10 158 (10.39%) Software Engineering
11 152 (9.99%) Continuous Integration
12 133 (8.74%) Artificial Intelligence
13 131 (8.61%) Scrum
14 130 (8.55%) Business Intelligence
15 120 (7.89%) Microservices
16 119 (7.82%) Computer Science
17 118 (7.76%) Problem-Solving
18 106 (6.97%) Visualisation
19 99 (6.51%) Migration
20 98 (6.44%) MapReduce
Programming Languages
1 902 (59.30%) Python
2 584 (38.40%) SQL
3 520 (34.19%) Java
4 493 (32.41%) Scala
5 149 (9.80%) R
6 69 (4.54%) JavaScript
7 66 (4.34%) C#
8 57 (3.75%) C++
9 53 (3.48%) Shell Script
10 50 (3.29%) Bash Shell
11 41 (2.70%) Java 8
12 40 (2.63%) Ruby
13 26 (1.71%) Go
14 25 (1.64%) C
14 25 (1.64%) MATLAB
14 25 (1.64%) T-SQL
15 23 (1.51%) PL/SQL
16 18 (1.18%) Perl
16 18 (1.18%) U-SQL
17 15 (0.99%) PowerShell
Qualifications
1 91 (5.98%) Degree
2 49 (3.22%) PhD
3 48 (3.16%) Security Cleared
4 38 (2.50%) SC Cleared
5 32 (2.10%) Computer Science Degree
6 27 (1.78%) AWS Certification
7 18 (1.18%) PMI Certification
7 18 (1.18%) PMP
8 17 (1.12%) Master's Degree
9 12 (0.79%) TOGAF Certification
10 10 (0.66%) Numerate Degree
11 5 (0.33%) BPSS Clearance
11 5 (0.33%) DV Cleared
12 4 (0.26%) ISTQB
13 1 (0.066%) CCA
13 1 (0.066%) CFA
13 1 (0.066%) Postgraduate
Quality Assurance & Compliance
1 67 (4.40%) Data Quality
2 20 (1.31%) GDPR
3 12 (0.79%) QA
4 3 (0.20%) IFRS
4 3 (0.20%) ISO/IEC 27001
5 1 (0.066%) EMIR
5 1 (0.066%) HL7
5 1 (0.066%) MiFID
5 1 (0.066%) PCI DSS
System Software
1 275 (18.08%) Docker
2 162 (10.65%) HDFS
3 82 (5.39%) Apache Flume
4 19 (1.25%) Apache ZooKeeper
5 17 (1.12%) Active Directory
6 4 (0.26%) Virtual Machines
7 3 (0.20%) VMware Infrastructure
8 1 (0.066%) NFS
8 1 (0.066%) Virtual Servers
Systems Management
1 180 (11.83%) Kubernetes
2 122 (8.02%) Ansible
3 100 (6.57%) Oozie
4 99 (6.51%) Hadoop YARN
5 88 (5.79%) Terraform
6 42 (2.76%) Puppet
7 36 (2.37%) Opscode Chef
8 18 (1.18%) Kibana
9 14 (0.92%) Kubeflow
10 13 (0.85%) logstash
11 10 (0.66%) Mesos
12 8 (0.53%) Marathon
13 6 (0.39%) Grafana
14 5 (0.33%) Ambari
15 4 (0.26%) Control-M
15 4 (0.26%) Docker Swarm
15 4 (0.26%) Graphite
15 4 (0.26%) Nagios
15 4 (0.26%) StatsD
16 3 (0.20%) rkt
Vendors
1 228 (14.99%) Google
2 138 (9.07%) Cloudera
3 101 (6.64%) Databricks
4 100 (6.57%) Oracle
5 88 (5.79%) Tableau
6 83 (5.46%) Informatica
7 76 (5.00%) Microsoft
8 61 (4.01%) Talend
9 53 (3.48%) Hortonworks
10 44 (2.89%) SAP
11 41 (2.70%) MapR
12 33 (2.17%) Splunk
13 26 (1.71%) IBM
14 19 (1.25%) IBM Sterling
15 15 (0.99%) New Relic
15 15 (0.99%) Teradata
16 13 (0.85%) Atlassian
17 11 (0.72%) Sybase
18 10 (0.66%) MicroStrategy
18 10 (0.66%) Red Hat