Period
to 10 December 2018

The following table provides summary statistics for contract job vacancies with a requirement for Spark SQL skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark SQL over the 6 months to 10 December 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark SQL
UK
6 months to
10 Dec 2018
Same period 2017 Same period 2016
Rank 842 871 864
Rank change year-on-year +29 -7 +22
Contract jobs citing Spark SQL 40 28 24
As % of all contract IT jobs advertised in the UK 0.038% 0.027% 0.023%
As % of the Database & Business Intelligence category 0.17% 0.12% 0.10%
Number of daily rates quoted 21 14 17
UK median daily rate £538 £650 £550
Median daily rate % change year-on-year -17.31% +18.18% -10.20%
10th Percentile £415 £588 £463
90th Percentile £688 £700 £650
UK excluding London median daily rate £500 £700 £550
% change year-on-year -28.57% +27.27% +10.00%
Number of hourly rates quoted 0 0 0
UK median hourly rate - - -
UK excluding London median hourly rate - - -

Spark SQL is in the Databases and Business Intelligence category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for database or business intelligence skills.

All Database and Business Intelligence Skills
UK
Contract vacancies with a requirement for database or business intelligence skills 23,738 22,517 23,206
As % of all contract IT jobs advertised in the UK 22.60% 21.97% 22.60%
Number of daily rates quoted 16,530 15,464 15,340
UK median daily rate £475 £450 £425
Median daily rate % change year-on-year +5.56% +5.88% +3.03%
10th Percentile £313 £310 £286
90th Percentile £638 £625 £600
UK excluding London median daily rate £425 £400 £380
% change year-on-year +6.25% +5.26% +1.33%
Number of hourly rates quoted 252 266 355
UK median hourly rate £32.49 £31.23 £28.00
Median hourly rate % change year-on-year +4.03% +11.54% -0.22%
10th Percentile £13.72 £14.25 £10.00
90th Percentile £57.50 £58.75 £51.84
UK excluding London median hourly rate £33.00 £35.00 £28.00
% change year-on-year -5.71% +25.00% -3.45%

Spark SQL
Job Vacancy Trend

Job postings citing Spark SQL as a percentage of all IT jobs advertised.

Job vacancy trend for Spark SQL in the UK

Spark SQL
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark SQL.

Contractor daily rate trend for Spark SQL in the UK

Spark SQL
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Spark SQL over the 6 months to 10 December 2018.

Contractor daily rate histogram for Spark SQL in the UK

Spark SQL
Top 7 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark SQL within the UK over the 6 months to 10 December 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -18 37 £563 -10.00% 2
London +22 27 £600 -4.00% 2
UK excluding London -25 12 £500 -28.57%
South East -22 6 £525 -
North of England - 3 £500 -
North East - 3 £500 -
Northern Ireland - 3 £434 -

For the 6 months to 10 December 2018, IT contractor jobs citing Spark SQL also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Spark SQL.

1 37 (92.50%) Apache Spark
2 36 (90.00%) SQL
3 32 (80.00%) Hadoop
4 31 (77.50%) Apache Hive
4 31 (77.50%) Python
5 30 (75.00%) Java
5 30 (75.00%) Big Data
6 29 (72.50%) Scala
7 28 (70.00%) Kafka
8 22 (55.00%) Agile Software Development
9 20 (50.00%) Sqoop
10 18 (45.00%) HBase
10 18 (45.00%) ETL
11 17 (42.50%) Analytics
12 16 (40.00%) Oracle
12 16 (40.00%) Linux
13 15 (37.50%) Apache
13 15 (37.50%) Cloudera
13 15 (37.50%) Impala
14 14 (35.00%) Hadoop YARN
14 14 (35.00%) MapReduce
14 14 (35.00%) Kanban
14 14 (35.00%) Finance
15 13 (32.50%) Unix
15 13 (32.50%) Elasticsearch
15 13 (32.50%) Relational Database
16 12 (30.00%) Spark Streaming
16 12 (30.00%) Data Modelling
16 12 (30.00%) Scrum
16 12 (30.00%) Git (software)

Spark SQL
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 37 (92.50%) Apache Spark
2 15 (37.50%) Apache
3 13 (32.50%) Elasticsearch
4 5 (12.50%) Apache Solr
5 3 (7.50%) Confluence
5 3 (7.50%) SAS
6 2 (5.00%) Apache Pig
7 1 (2.50%) Apache Airflow
7 1 (2.50%) Gunicorn
7 1 (2.50%) Jupyter
7 1 (2.50%) nginx
Cloud Services
1 9 (22.50%) Google Cloud Platform
2 8 (20.00%) Amazon AWS
2 8 (20.00%) GitHub
3 5 (12.50%) Microsoft Azure
4 3 (7.50%) Cloud Computing
5 1 (2.50%) Amazon EC2
5 1 (2.50%) Amazon EMR
5 1 (2.50%) Amazon S3
5 1 (2.50%) Azure Data Factory
5 1 (2.50%) Virtual Private Cloud
Communications & Networking
1 3 (7.50%) SNA
2 1 (2.50%) SSH
Database & Business Intelligence
1 32 (80.00%) Hadoop
2 31 (77.50%) Apache Hive
3 30 (75.00%) Big Data
4 20 (50.00%) Sqoop
5 18 (45.00%) HBase
6 15 (37.50%) Impala
7 13 (32.50%) Relational Database
8 10 (25.00%) Data Lake
9 9 (22.50%) Apache Cassandra
10 8 (20.00%) Data Warehouse
11 7 (17.50%) Presto
12 6 (15.00%) MongoDB
13 5 (12.50%) NoSQL
13 5 (12.50%) Oracle BI EE
13 5 (12.50%) Power BI
13 5 (12.50%) Qlik Sense
14 4 (10.00%) SQL Server
15 3 (7.50%) Amazon Redshift
15 3 (7.50%) CouchDB
15 3 (7.50%) Enterprise Data Warehouse
Development Applications
1 12 (30.00%) Git (software)
2 11 (27.50%) Jenkins
2 11 (27.50%) JIRA
3 3 (7.50%) Bitbucket
4 2 (5.00%) SoapUI
5 1 (2.50%) Log4j
5 1 (2.50%) ProC
General
1 14 (35.00%) Finance
2 8 (20.00%) Publishing
3 6 (15.00%) Banking
4 1 (2.50%) Investment Banking
Job Titles
1 14 (35.00%) Developer
2 11 (27.50%) Big Data Engineer
2 11 (27.50%) Data Engineer
3 7 (17.50%) Big Data Developer
3 7 (17.50%) Consultant
4 5 (12.50%) Big Data Consultant
4 5 (12.50%) Data Consultant
5 4 (10.00%) Architect
5 4 (10.00%) Scala Developer
6 3 (7.50%) Ab Initio Developer
6 3 (7.50%) Big Data Architect
6 3 (7.50%) Cloud Engineer
6 3 (7.50%) Data Architect
6 3 (7.50%) Hadoop Engineer
6 3 (7.50%) Technical Consultant
7 2 (5.00%) Apache Spark Developer
7 2 (5.00%) Big Data Specialist
7 2 (5.00%) Hadoop Developer
7 2 (5.00%) Python Engineer
8 1 (2.50%) E-Commerce Consultant
Libraries, Frameworks & Software Standards
1 28 (70.00%) Kafka
2 12 (30.00%) Spark Streaming
3 8 (20.00%) RESTful
4 5 (12.50%) AngularJS
4 5 (12.50%) D3.js
4 5 (12.50%) Elastic Stack
4 5 (12.50%) HTML
4 5 (12.50%) MLlib
4 5 (12.50%) Oracle GoldenGate
4 5 (12.50%) Pandas
5 3 (7.50%) ActiveMQ
5 3 (7.50%) CSS
5 3 (7.50%) Lucene
5 3 (7.50%) Node.js
5 3 (7.50%) React
5 3 (7.50%) Twitter Bootstrap
6 2 (5.00%) GraphX
6 2 (5.00%) HTML5
6 2 (5.00%) SOAP
6 2 (5.00%) Web Services
Miscellaneous
1 3 (7.50%) Data Centre
2 2 (5.00%) Social Network
3 1 (2.50%) Algorithms
3 1 (2.50%) Analytical Skills
3 1 (2.50%) Equities
3 1 (2.50%) IBM Mainframe
3 1 (2.50%) Mainframe
3 1 (2.50%) Virtual Team
Operating Systems
1 16 (40.00%) Linux
2 13 (32.50%) Unix
3 1 (2.50%) Windows
Processes & Methodologies
1 22 (55.00%) Agile Software Development
2 18 (45.00%) ETL
3 17 (42.50%) Analytics
4 14 (35.00%) Kanban
4 14 (35.00%) MapReduce
5 12 (30.00%) Data Modelling
5 12 (30.00%) Scrum
5 12 (30.00%) Visualisation
6 10 (25.00%) Data Engineering
6 10 (25.00%) Data Ingestion
6 10 (25.00%) DevOps
7 9 (22.50%) Continuous Integration
7 9 (22.50%) Data Science
7 9 (22.50%) Data Strategy
8 8 (20.00%) Data Visualisation
8 8 (20.00%) Deployment Automation
8 8 (20.00%) Requirements Management
9 7 (17.50%) Microservices
9 7 (17.50%) Performance Tuning
9 7 (17.50%) SDLC
Programming Languages
1 36 (90.00%) SQL
2 31 (77.50%) Python
3 30 (75.00%) Java
4 29 (72.50%) Scala
5 6 (15.00%) R
6 5 (12.50%) Go
7 4 (10.00%) Shell Script
8 3 (7.50%) JavaScript
9 1 (2.50%) Bash Shell
9 1 (2.50%) C
9 1 (2.50%) C++
Qualifications
1 1 (2.50%) Computer Science Degree
1 1 (2.50%) Degree
Quality Assurance & Compliance
1 4 (10.00%) QA
System Software
1 12 (30.00%) Apache Flume
2 11 (27.50%) HDFS
3 6 (15.00%) Docker
4 1 (2.50%) Apache ZooKeeper
Systems Management
1 14 (35.00%) Hadoop YARN
2 10 (25.00%) Ansible
3 6 (15.00%) Control-M
3 6 (15.00%) ITRS Geneos
4 5 (12.50%) Kubernetes
5 4 (10.00%) Kibana
5 4 (10.00%) logstash
6 3 (7.50%) Oozie
7 1 (2.50%) Grafana
7 1 (2.50%) Packer
7 1 (2.50%) Prometheus
7 1 (2.50%) Terraform
7 1 (2.50%) Tivoli
7 1 (2.50%) ZABBIX
Vendors
1 16 (40.00%) Oracle
2 15 (37.50%) Cloudera
3 11 (27.50%) Google
4 9 (22.50%) Tableau
5 3 (7.50%) Ab Initio
5 3 (7.50%) IBM
5 3 (7.50%) Palantir
5 3 (7.50%) Talend
6 1 (2.50%) Atlassian
6 1 (2.50%) CA Agile Central
6 1 (2.50%) Greenplum
6 1 (2.50%) Hortonworks
6 1 (2.50%) Netezza
6 1 (2.50%) Solace
6 1 (2.50%) Teradata
6 1 (2.50%) TIBCO