Period
to 21 February 2020

The following table provides summary statistics for contract job vacancies with a requirement for Spark SQL skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark SQL over the 6 months to 21 February 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark SQL
UK
6 months to
21 Feb 2020
Same period 2019 Same period 2018
Rank 654 845 865
Rank change year-on-year +191 +20 -25
Contract jobs citing Spark SQL 78 41 23
As % of all contract IT jobs advertised in the UK 0.11% 0.041% 0.023%
As % of the Database & Business Intelligence category 0.54% 0.19% 0.10%
Number of daily rates quoted 61 29 13
UK median daily rate £575 £550 £625
Median daily rate % change year-on-year +4.55% -12.00% +16.28%
10th Percentile £438 £410 £588
90th Percentile £725 £726 £700
UK excluding London median daily rate £575 £525 -
% change year-on-year +9.52% - -
Number of hourly rates quoted 0 0 0
UK median hourly rate - - -
UK excluding London median hourly rate - - -

Spark SQL is in the Databases and Business Intelligence category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for database or business intelligence skills.

All Database and Business Intelligence Skills
UK
Contract vacancies with a requirement for database or business intelligence skills 14,531 22,079 22,152
As % of all contract IT jobs advertised in the UK 21.23% 22.28% 21.81%
Number of daily rates quoted 9,600 15,224 15,426
UK median daily rate £475 £475 £463
Median daily rate % change year-on-year - +2.70% +8.82%
10th Percentile £319 £320 £313
90th Percentile £663 £650 £625
UK excluding London median daily rate £438 £425 £400
% change year-on-year +2.94% +6.25% +1.27%
Number of hourly rates quoted 240 253 252
UK median hourly rate £31.25 £35.00 £34.75
Median hourly rate % change year-on-year -10.71% +0.72% +28.70%
10th Percentile £13.97 £13.05 £15.02
90th Percentile £57.50 £55.80 £56.64
UK excluding London median hourly rate £26.25 £33.00 £38.25
% change year-on-year -20.45% -13.73% +56.12%

Spark SQL
Job Vacancy Trend

Job postings citing Spark SQL as a percentage of all IT jobs advertised.

Job vacancy trend for Spark SQL in the UK

Spark SQL
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark SQL.

Contractor daily rate trend for Spark SQL in the UK

Spark SQL
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Spark SQL over the 6 months to 21 February 2020.

Contractor daily rate histogram for Spark SQL in the UK

Spark SQL
Top 11 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark SQL within the UK over the 6 months to 21 February 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +177 54 £563 +3.45% 6
London +160 47 £588 +2.17% 1
UK excluding London +127 31 £575 +9.52% 6
Scotland - 24 £575 - 1
North of England +69 3 £438 -12.50%
Midlands +64 2 £513 +34.87% 1
East Midlands +39 2 £513 +34.87%
Yorkshire - 2 £438 -
South East +80 1 - - 1
South West - 1 - - 3
North West - 1 - -

For the 6 months to 21 February 2020, IT contractor jobs citing Spark SQL also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Spark SQL.

1 71 (91.03%) Apache Spark
2 64 (82.05%) Big Data
3 58 (74.36%) Scala
4 56 (71.79%) Hadoop
5 54 (69.23%) Apache Hive
6 52 (66.67%) SQL
7 49 (62.82%) Impala
8 42 (53.85%) Kafka
9 39 (50.00%) Python
10 38 (48.72%) HDFS
11 36 (46.15%) Agile Software Development
12 27 (34.62%) Banking
12 27 (34.62%) HBase
12 27 (34.62%) Data Development
13 25 (32.05%) Finance
14 24 (30.77%) ETL
15 23 (29.49%) Java
16 19 (24.36%) Investment Banking
17 18 (23.08%) Cloudera
18 16 (20.51%) Hortonworks
19 15 (19.23%) PySpark
19 15 (19.23%) AWS
19 15 (19.23%) Scrum
20 14 (17.95%) Problem-Solving
21 13 (16.67%) Continuous Integration
22 12 (15.38%) Software Engineering
22 12 (15.38%) PostgreSQL
22 12 (15.38%) MySQL
22 12 (15.38%) SQL Server
23 11 (14.10%) SDLC

Spark SQL
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 71 (91.03%) Apache Spark
2 7 (8.97%) Jupyter
3 5 (6.41%) Apache Airflow
3 5 (6.41%) Confluence
4 3 (3.85%) Apache Solr
4 3 (3.85%) SAS
5 2 (2.56%) Elasticsearch
6 1 (1.28%) Apache Pig
Applications
1 6 (7.69%) Microsoft Excel
Cloud Services
1 15 (19.23%) AWS
2 9 (11.54%) Azure
3 8 (10.26%) Power Platform
4 7 (8.97%) Amazon Kinesis
4 7 (8.97%) GCP
5 5 (6.41%) Amazon EMR
6 4 (5.13%) AWS CloudFormation
7 2 (2.56%) Amazon S3
7 2 (2.56%) GitHub
8 1 (1.28%) Azure Data Factory
8 1 (1.28%) OpenShift
Communications & Networking
1 1 (1.28%) Kerberos
Database & Business Intelligence
1 64 (82.05%) Big Data
2 56 (71.79%) Hadoop
3 54 (69.23%) Apache Hive
4 49 (62.82%) Impala
5 27 (34.62%) HBase
6 12 (15.38%) MySQL
6 12 (15.38%) PostgreSQL
6 12 (15.38%) SQL Server
7 8 (10.26%) Power BI
7 8 (10.26%) Sqoop
8 4 (5.13%) Amazon Athena
8 4 (5.13%) Amazon Redshift
9 2 (2.56%) Apache Cassandra
9 2 (2.56%) Data Lake
9 2 (2.56%) Neo4j
9 2 (2.56%) NoSQL
10 1 (1.28%) Data Warehouse
10 1 (1.28%) Relational Database
10 1 (1.28%) SQL Server Integration Services
10 1 (1.28%) SQL Server Reporting Services
Development Applications
1 5 (6.41%) Git (software)
1 5 (6.41%) JIRA
2 2 (2.56%) JUnit
3 1 (1.28%) Gradle
3 1 (1.28%) IBM UrbanCode
3 1 (1.28%) Jenkins
3 1 (1.28%) Maven
3 1 (1.28%) Sonatype Nexus
General
1 27 (34.62%) Banking
2 25 (32.05%) Finance
3 19 (24.36%) Investment Banking
4 7 (8.97%) Front Office
5 2 (2.56%) Retail
6 1 (1.28%) Telecoms
Job Titles
1 47 (60.26%) Developer
2 26 (33.33%) Data Engineer
3 21 (26.92%) Scala Developer
4 16 (20.51%) Apache Spark Developer
5 14 (17.95%) Big Data Engineer
6 12 (15.38%) Python Developer
7 11 (14.10%) Big Data Developer
8 7 (8.97%) Hadoop Developer
9 6 (7.69%) Python Engineer
10 4 (5.13%) Java Developer
10 4 (5.13%) SQL Developer
11 3 (3.85%) Architect
11 3 (3.85%) Data Architect
11 3 (3.85%) Hadoop Engineer
11 3 (3.85%) Senior Developer
12 2 (2.56%) Database Developer
12 2 (2.56%) Junior
12 2 (2.56%) Junior Data Engineer
12 2 (2.56%) Senior Scala Developer
13 1 (1.28%) Analyst
Libraries, Frameworks & Software Standards
1 42 (53.85%) Kafka
2 15 (19.23%) PySpark
3 9 (11.54%) Spark Streaming
4 4 (5.13%) Pandas
5 2 (2.56%) Apache NiFi
5 2 (2.56%) Lucene
5 2 (2.56%) Web Services
6 1 (1.28%) XML
Miscellaneous
1 7 (8.97%) Analytical Skills
1 7 (8.97%) Greenfield Project
2 2 (2.56%) Algorithms
Operating Systems
1 3 (3.85%) Unix
2 1 (1.28%) Linux
Processes & Methodologies
1 36 (46.15%) Agile Software Development
2 27 (34.62%) Data Development
3 24 (30.77%) ETL
4 15 (19.23%) Scrum
5 14 (17.95%) Problem-Solving
6 13 (16.67%) Continuous Integration
7 12 (15.38%) Software Engineering
8 11 (14.10%) DevOps
8 11 (14.10%) SDLC
9 9 (11.54%) Acceptance Criteria
10 8 (10.26%) Business Intelligence
10 8 (10.26%) Data Engineering
11 7 (8.97%) Capital Modelling
11 7 (8.97%) Data Analysis
11 7 (8.97%) Data Management
11 7 (8.97%) Data Visualisation
11 7 (8.97%) Functional Programming
11 7 (8.97%) Use Case
11 7 (8.97%) Visualisation
12 6 (7.69%) Analytical Thinking
Programming Languages
1 58 (74.36%) Scala
2 52 (66.67%) SQL
3 39 (50.00%) Python
4 23 (29.49%) Java
5 8 (10.26%) Bash Shell
6 7 (8.97%) R
7 1 (1.28%) Elm
7 1 (1.28%) Haskell
7 1 (1.28%) JavaScript
Qualifications
1 2 (2.56%) Degree
Quality Assurance & Compliance
1 6 (7.69%) Data Quality
2 4 (5.13%) MiFID
3 1 (1.28%) IFRS
3 1 (1.28%) QA
System Software
1 38 (48.72%) HDFS
2 1 (1.28%) Apache Flume
2 1 (1.28%) Apache ZooKeeper
Systems Management
1 11 (14.10%) Hadoop YARN
2 4 (5.13%) Oozie
3 1 (1.28%) Ambari
Vendors
1 18 (23.08%) Cloudera
2 16 (20.51%) Hortonworks
3 9 (11.54%) Microsoft
4 7 (8.97%) Foundry
4 7 (8.97%) Google
4 7 (8.97%) Oracle
4 7 (8.97%) Palantir
4 7 (8.97%) Tableau
5 4 (5.13%) GoldenSource
6 2 (2.56%) StreamSets
7 1 (1.28%) Splunk