Period
to 23 April 2021

The following table provides summary statistics for contract job vacancies with a requirement for HDFS skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited HDFS over the 6 months to 23 April 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Hadoop Distributed File System (HDFS)
UK
6 months to
23 Apr 2021
Same period 2020 Same period 2019
Rank 489 519 569
Rank change year-on-year +30 +50 -6
Contract jobs citing HDFS 171 136 320
As % of all contract jobs advertised in the UK 0.32% 0.24% 0.34%
As % of the System Software category 2.18% 1.56% 2.20%
Number of daily rates quoted 103 98 192
10th Percentile £356 £436 £463
25th Percentile £456 £478 £508
Median daily rate (50th Percentile) £550 £557 £575
Median % change year-on-year -1.26% -3.13% -
75th Percentile £600 £634 £650
90th Percentile £625 £700 £699
UK excluding London median daily rate £503 £525 £523
% change year-on-year -4.29% +0.48% -9.13%
Number of hourly rates quoted 1 0 0
Median hourly rate £14.50 - -
UK excluding London median hourly rate £14.50 - -

HDFS is in the System Software category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for system software skills.

All System Software Skills
UK
Contract vacancies with a requirement for system software skills 7,845 8,717 14,564
As % of all contract IT jobs advertised in the UK 14.50% 15.52% 15.29%
Number of daily rates quoted 5,411 5,869 9,824
10th Percentile £235 £168 £180
25th Percentile £375 £325 £350
Median daily rate (50th Percentile) £483 £450 £475
Median % change year-on-year +7.22% -5.26% +5.56%
75th Percentile £575 £563 £575
90th Percentile £650 £638 £638
UK excluding London median daily rate £415 £425 £425
% change year-on-year -2.35% - +6.25%
Number of hourly rates quoted 284 475 645
10th Percentile £11.63 £11.26 £10.92
25th Percentile £13.91 £13.45 £12.76
Median hourly rate £20.00 £17.00 £16.00
Median % change year-on-year +17.65% +6.25% +0.88%
75th Percentile £47.81 £22.63 £25.67
90th Percentile £60.18 £47.83 £47.50
UK excluding London median hourly rate £17.93 £16.50 £15.37
% change year-on-year +8.68% +7.35% +2.47%

HDFS
Job Vacancy Trend

Job postings citing HDFS as a proportion of all IT jobs advertised.

Job vacancy trend for HDFS in the UK

HDFS
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing HDFS.

Daily rate trend for HDFS in the UK

HDFS
Daily Rate Histogram

Daily rate distribution for jobs citing HDFS over the 6 months to 23 April 2021.

Daily rate histogram for HDFS in the UK

HDFS
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing HDFS.

Hourly rate trend for HDFS in the UK

HDFS
Top 13 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing HDFS within the UK over the 6 months to 23 April 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +89 144 £540 -3.05% 13
London +23 70 £575 -4.17% 3
UK excluding London +75 68 £503 -4.29% 10
Work from Home - 50 £555 - 1
South East +71 26 £475 -5.94% 3
North of England +61 12 £555 +4.72% 1
Midlands +35 11 £521 +44.79% 3
West Midlands +31 11 £521 +44.79% 3
South West +40 9 £555 +5.21%
East of England +39 8 £338 - 2
Yorkshire +35 7 £555 +7.77%
North West +34 5 £356 -52.53% 1
Wales +2 2 £475 - 1

For the 6 months to 23 April 2021, IT contractor jobs citing HDFS also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for HDFS.

1 162 (94.74%) Hadoop
2 149 (87.13%) Apache Hive
3 128 (74.85%) Apache Spark
4 100 (58.48%) Big Data
5 87 (50.88%) Scala
6 81 (47.37%) SQL
7 74 (43.27%) ETL
8 71 (41.52%) Cloudera
8 71 (41.52%) Impala
9 69 (40.35%) Agile Software Development
10 67 (39.18%) HBase
11 65 (38.01%) Python
12 64 (37.43%) Apache
13 62 (36.26%) Security Cleared
14 60 (35.09%) Jenkins
15 59 (34.50%) Java
16 57 (33.33%) Linux
17 56 (32.75%) SC Cleared
18 55 (32.16%) Kafka
19 53 (30.99%) DevOps
20 49 (28.65%) Git (software)
21 47 (27.49%) Oozie
22 46 (26.90%) Scrum
23 42 (24.56%) AWS
23 42 (24.56%) Amazon S3
23 42 (24.56%) Ansible
24 40 (23.39%) Data Ingestion
25 37 (21.64%) Data Modelling
26 36 (21.05%) Relational Database
27 35 (20.47%) Hadoop YARN

HDFS
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 128 (74.85%) Apache Spark
2 64 (37.43%) Apache
3 34 (19.88%) Confluence
4 15 (8.77%) Elasticsearch
5 8 (4.68%) Apache Airflow
6 6 (3.51%) Apache Solr
7 5 (2.92%) Apache Pig
8 4 (2.34%) SAS
Applications
1 18 (10.53%) Spreadsheet
2 3 (1.75%) Microsoft Excel
Cloud Services
1 42 (24.56%) Amazon S3
1 42 (24.56%) AWS
2 34 (19.88%) Azure
3 19 (11.11%) Amazon EMR
4 12 (7.02%) Power Platform
5 11 (6.43%) GitHub
6 7 (4.09%) Azure Active Directory
6 7 (4.09%) GCP
6 7 (4.09%) Microsoft 365
7 5 (2.92%) Amazon EC2
7 5 (2.92%) AWS CloudFormation
8 4 (2.34%) Amazon CloudWatch
8 4 (2.34%) AWS Lambda
8 4 (2.34%) SaaS
8 4 (2.34%) Serverless
9 3 (1.75%) Amazon ElastiCache
9 3 (1.75%) Amazon Kinesis
9 3 (1.75%) Azure Data Factory
9 3 (1.75%) PaaS
10 2 (1.17%) OpenShift
Communications & Networking
1 7 (4.09%) Ethernet
1 7 (4.09%) Firewall
1 7 (4.09%) Kerberos
1 7 (4.09%) VLAN
1 7 (4.09%) VSAN
1 7 (4.09%) WAN
2 3 (1.75%) Network Security
3 2 (1.17%) SMTP
3 2 (1.17%) SSL
4 1 (0.58%) FTP
Database & Business Intelligence
1 162 (94.74%) Hadoop
2 149 (87.13%) Apache Hive
3 100 (58.48%) Big Data
4 71 (41.52%) Impala
5 67 (39.18%) HBase
6 36 (21.05%) Relational Database
7 29 (16.96%) NoSQL
8 25 (14.62%) Data Lake
8 25 (14.62%) MySQL
8 25 (14.62%) Pentaho
9 22 (12.87%) Spark SQL
10 20 (11.70%) Data Warehouse
11 14 (8.19%) Apache Cassandra
12 13 (7.60%) Parquet
13 12 (7.02%) PostgreSQL
13 12 (7.02%) Power BI
13 12 (7.02%) Sqoop
14 11 (6.43%) SQL Server
14 11 (6.43%) Tableau
15 10 (5.85%) MongoDB
Development Applications
1 60 (35.09%) Jenkins
2 49 (28.65%) Git (software)
3 35 (20.47%) JIRA
4 28 (16.37%) Sonatype Nexus
5 25 (14.62%) IBM UrbanCode
6 23 (13.45%) GitLab
7 20 (11.70%) Maven
8 12 (7.02%) Gradle
9 10 (5.85%) Artifactory
10 9 (5.26%) Bitbucket
11 5 (2.92%) Cucumber
11 5 (2.92%) JUnit
12 4 (2.34%) Atlassian Bamboo
13 2 (1.17%) Eclipse
13 2 (1.17%) Oracle APEX
13 2 (1.17%) Subversion
14 1 (0.58%) ER Studio
14 1 (0.58%) ERwin
14 1 (0.58%) kappa
14 1 (0.58%) TeamCity
General
1 34 (19.88%) Finance
2 18 (10.53%) Banking
2 18 (10.53%) Public Sector
3 8 (4.68%) Investment Banking
4 5 (2.92%) French Language
5 2 (1.17%) Games
6 1 (0.58%) Retail
6 1 (0.58%) Telecoms
Job Titles
1 57 (33.33%) Developer
2 48 (28.07%) Data Engineer
3 25 (14.62%) Big Data Developer
4 24 (14.04%) Apache Spark Developer
4 24 (14.04%) Scala Developer
5 21 (12.28%) Hadoop Developer
6 16 (9.36%) Big Data Engineer
7 14 (8.19%) Architect
8 10 (5.85%) Scala Engineer
9 9 (5.26%) Software Engineer
10 8 (4.68%) Analyst
10 8 (4.68%) Cloud Engineer
10 8 (4.68%) Data Architect
11 7 (4.09%) Cloud Developer
11 7 (4.09%) Hadoop Engineer
11 7 (4.09%) Python Developer
12 6 (3.51%) Infrastructure Engineer
12 6 (3.51%) Python Engineer
12 6 (3.51%) Solutions Architect
13 5 (2.92%) Pentaho Developer
Libraries, Frameworks & Software Standards
1 55 (32.16%) Kafka
2 18 (10.53%) PySpark
3 16 (9.36%) Apache NiFi
4 13 (7.60%) JSON
4 13 (7.60%) Spark Streaming
5 11 (6.43%) XML
6 10 (5.85%) RabbitMQ
7 9 (5.26%) Spring
8 6 (3.51%) Anaconda
9 5 (2.92%) SAML
10 4 (2.34%) Akka
11 3 (1.75%) GraphX
11 3 (1.75%) Lucene
11 3 (1.75%) MLlib
12 1 (0.58%) AngularJS
12 1 (0.58%) HTML
12 1 (0.58%) React
12 1 (0.58%) Web Services
12 1 (0.58%) YAML
Miscellaneous
1 25 (14.62%) Linux Command Line
2 18 (10.53%) Analytical Skills
3 6 (3.51%) VSAM
4 5 (2.92%) Data Structures
5 4 (2.34%) Analytical Mindset
6 3 (1.75%) Cloud Native
7 2 (1.17%) Private Cloud
8 1 (0.58%) Clustering
8 1 (0.58%) Cybercrime
8 1 (0.58%) Greenfield Project
8 1 (0.58%) Management Information System
8 1 (0.58%) Organisational Skills
8 1 (0.58%) Reference Data
8 1 (0.58%) Virtual Team
Operating Systems
1 57 (33.33%) Linux
2 14 (8.19%) Windows
3 11 (6.43%) Unix
4 7 (4.09%) Ubuntu
5 2 (1.17%) Red Hat Enterprise Linux
6 1 (0.58%) CentOS
Processes & Methodologies
1 74 (43.27%) ETL
2 69 (40.35%) Agile Software Development
3 53 (30.99%) DevOps
4 46 (26.90%) Scrum
5 40 (23.39%) Data Ingestion
6 37 (21.64%) Data Modelling
7 32 (18.71%) Data Engineering
8 29 (16.96%) Configuration Management
9 28 (16.37%) ETL Development
9 28 (16.37%) Test Automation
10 26 (15.20%) Problem-Solving
11 25 (14.62%) Analytics
11 25 (14.62%) Data Integration
11 25 (14.62%) Data Management
11 25 (14.62%) Kanban
12 24 (14.04%) Continuous Integration
13 23 (13.45%) Data Acquisition
14 22 (12.87%) Software Engineering
15 20 (11.70%) Unit Testing
16 19 (11.11%) TDD
Programming Languages
1 87 (50.88%) Scala
2 81 (47.37%) SQL
3 65 (38.01%) Python
4 59 (34.50%) Java
5 29 (16.96%) Shell Script
6 11 (6.43%) Elm
6 11 (6.43%) Haskell
7 6 (3.51%) R
8 5 (2.92%) Bash Shell
9 4 (2.34%) PL/SQL
10 3 (1.75%) C
10 3 (1.75%) HiveQL
10 3 (1.75%) Perl
10 3 (1.75%) VBA
11 2 (1.17%) Go
11 2 (1.17%) JavaScript
11 2 (1.17%) Ruby
11 2 (1.17%) T-SQL
12 1 (0.58%) COBOL
12 1 (0.58%) Groovy
Qualifications
1 62 (36.26%) Security Cleared
2 56 (32.75%) SC Cleared
3 5 (2.92%) Degree
4 4 (2.34%) Computer Science Degree
5 3 (1.75%) AWS Certification
6 1 (0.58%) BPSS Clearance
Quality Assurance & Compliance
1 9 (5.26%) QA
2 6 (3.51%) Data Quality
3 2 (1.17%) GDPR
4 1 (0.58%) PCI DSS
System Software
1 17 (9.94%) Docker
2 13 (7.60%) Apache ZooKeeper
3 11 (6.43%) Active Directory
4 7 (4.09%) VMware NSX
4 7 (4.09%) vSphere
5 4 (2.34%) Apache Flume
6 2 (1.17%) Terminal Services
Systems Management
1 47 (27.49%) Oozie
2 42 (24.56%) Ansible
3 35 (20.47%) Hadoop YARN
4 17 (9.94%) Kubernetes
5 9 (5.26%) Ambari
5 9 (5.26%) Puppet
6 7 (4.09%) Kibana
6 7 (4.09%) Microsoft Intune
7 6 (3.51%) Grafana
7 6 (3.51%) vCenter Server
8 3 (1.75%) Terraform
9 2 (1.17%) Apache Slider
9 2 (1.17%) Control-M
9 2 (1.17%) Red Hat Satellite
9 2 (1.17%) SCCM
10 1 (0.58%) Chronos
10 1 (0.58%) Marathon
10 1 (0.58%) Mesos
Vendors
1 71 (41.52%) Cloudera
2 29 (16.96%) Oracle
3 13 (7.60%) Microsoft
4 10 (5.85%) Informatica
4 10 (5.85%) MapR
5 8 (4.68%) Hortonworks
6 7 (4.09%) Ab Initio
6 7 (4.09%) StreamSets
7 6 (3.51%) Cisco
7 6 (3.51%) Dell
7 6 (3.51%) EMC
7 6 (3.51%) Salesforce.com
7 6 (3.51%) Teradata
8 5 (2.92%) Databricks
9 4 (2.34%) Atlassian
9 4 (2.34%) Google
10 3 (1.75%) IBM
10 3 (1.75%) Talend
10 3 (1.75%) Vertica
11 2 (1.17%) CA