Period
to 23 September 2021

The following table provides summary statistics for contract job vacancies advertised in England with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 23 September 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Data Pipelines
UK > England
6 months to
23 Sep 2021
Same period 2020 Same period 2019
Rank 266 263 395
Rank change year-on-year -3 +132 +223
Contract jobs citing Data Pipeline 673 235 497
As % of all contract jobs advertised in England 1.05% 0.91% 0.62%
As % of the Processes & Methodologies category 1.17% 1.04% 0.70%
Number of daily rates quoted 431 154 401
10th Percentile £400 £413 £413
25th Percentile £461 £463 £463
Median daily rate (50th Percentile) £550 £550 £550
75th Percentile £638 £645 £613
90th Percentile £700 £688 £688
UK median daily rate £525 £550 £550
% change year-on-year -4.55% - +4.76%
Number of hourly rates quoted 10 3 5
10th Percentile £18.59 £39.09 £44.34
25th Percentile £25.77 £40.73 £47.43
Median hourly rate £36.25 £46.93 £49.13
Median % change year-on-year -22.75% -4.49% +15.60%
75th Percentile £48.88 £55.82 £52.29
90th Percentile £57.93 £59.08 £62.92
UK median hourly rate £37.50 £46.93 £49.13
% change year-on-year -20.09% -4.49% +15.60%

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in England with a requirement for process or methodology skills.

All Process and Methodology Skills
England
Contract vacancies with a requirement for process or methodology skills 57,368 22,631 70,578
As % of all contract IT jobs advertised in England 89.23% 87.52% 87.70%
Number of daily rates quoted 40,084 15,441 47,696
10th Percentile £315 £300 £300
25th Percentile £413 £392 £381
Median daily rate (50th Percentile) £513 £480 £475
Median % change year-on-year +6.77% +1.05% +2.70%
75th Percentile £613 £588 £575
90th Percentile £700 £675 £650
UK median daily rate £500 £475 £475
% change year-on-year +5.26% - +5.56%
Number of hourly rates quoted 1,509 636 1,911
10th Percentile £12.00 £12.01 £11.49
25th Percentile £14.55 £15.93 £14.33
Median hourly rate £24.30 £26.00 £22.93
Median % change year-on-year -6.54% +13.39% +14.65%
75th Percentile £48.75 £46.00 £46.45
90th Percentile £62.50 £55.00 £53.75
UK median hourly rate £24.50 £25.00 £22.63
% change year-on-year -2.00% +10.50% +16.38%

Data Pipeline
Job Vacancy Trend in England

Job postings citing Data Pipeline as a proportion of all IT jobs advertised in England.

Job vacancy trend for Data Pipeline in England

Data Pipeline
Contractor Daily Rate Trend in England

3-month moving average daily rate quoted in jobs citing Data Pipeline in England.

Daily rate trend for Data Pipeline in England

Data Pipeline
Daily Rate Histogram in England

Daily rate distribution for jobs citing Data Pipeline in England over the 6 months to 23 September 2021.

Daily rate histogram for Data Pipeline in England

Data Pipeline
Contractor Hourly Rate Trend in England

3-month moving average hourly rates quoted in jobs citing Data Pipeline in England.

Hourly rate trend for Data Pipeline in England

Data Pipeline
Hourly Rate Histogram in England

Hourly rate distribution of jobs citing Data Pipeline in England over the 6 months to 23 September 2021.

Hourly rate histogram for Data Pipeline in England

Data Pipeline
Contract Job Locations in England

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the England region over the 6 months to 23 September 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London -16 454 £550 - 130
South East -33 82 £538 - 14
North of England -47 43 £481 -12.50% 20
Yorkshire -25 28 £475 -5.00% 5
Midlands -59 28 £463 - 16
South West -25 26 £500 -11.11% 12
West Midlands - 19 £500 - 14
North West -41 15 £500 -9.09% 13
East of England -45 10 £478 -16.96% 5
East Midlands -37 7 £425 - 2
Data Pipeline
UK

For the 6 months to 23 September 2021, IT contractor jobs citing Data Pipeline also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the England region with a requirement for Data Pipeline.

1 402 (59.73%) Python
2 354 (52.60%) SQL
3 265 (39.38%) Analytics
4 263 (39.08%) AWS
5 260 (38.63%) Azure
6 235 (34.92%) Apache Spark
7 212 (31.50%) ETL
8 197 (29.27%) Data Warehouse
9 194 (28.83%) Data Engineering
10 192 (28.53%) Agile Software Development
11 185 (27.49%) Big Data
12 171 (25.41%) Kafka
12 171 (25.41%) DevOps
13 168 (24.96%) Java
14 157 (23.33%) Data Science
15 155 (23.03%) Data Modelling
16 147 (21.84%) Business Intelligence
17 144 (21.40%) Scala
17 144 (21.40%) Data Lake
18 141 (20.95%) Databricks
19 134 (19.91%) GCP
19 134 (19.91%) Hadoop
19 134 (19.91%) Finance
20 121 (17.98%) Machine Learning
21 117 (17.38%) Social Skills
22 113 (16.79%) NoSQL
23 109 (16.20%) Git (software)
24 105 (15.60%) Visualisation
24 105 (15.60%) Power Platform
24 105 (15.60%) Apache Airflow

Data Pipeline
Co-occurring IT Skills in England by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 235 (34.92%) Apache Spark
2 105 (15.60%) Apache Airflow
3 63 (9.36%) Apache
4 30 (4.46%) Luigi
5 11 (1.63%) Elasticsearch
5 11 (1.63%) Jupyter
6 10 (1.49%) Apache Pig
6 10 (1.49%) SharePoint
7 7 (1.04%) SAS
8 6 (0.89%) Apache Pulsar
8 6 (0.89%) Tomcat
9 5 (0.74%) CMS
9 5 (0.74%) Mule
9 5 (0.74%) Oracle Workflow
10 2 (0.30%) Confluence
10 2 (0.30%) Hyperledger
10 2 (0.30%) nginx
11 1 (0.15%) OpenStack
11 1 (0.15%) Pachyderm
Applications
1 12 (1.78%) Microsoft Excel
2 3 (0.45%) Microsoft PowerPoint
2 3 (0.45%) SPSS
3 1 (0.15%) Microsoft Office
3 1 (0.15%) SPSS Modeler
3 1 (0.15%) Stata
Business Applications
1 6 (0.89%) Dynamics CRM
2 3 (0.45%) Dynamics AX
2 3 (0.45%) Markit EDM
3 2 (0.30%) Dynamics GP
4 1 (0.15%) Adobe Campaign
4 1 (0.15%) Aligne
4 1 (0.15%) Oracle Procure-to-Pay
4 1 (0.15%) SAP R/3
Cloud Services
1 263 (39.08%) AWS
2 260 (38.63%) Azure
3 134 (19.91%) GCP
4 105 (15.60%) Power Platform
5 90 (13.37%) Serverless
6 85 (12.63%) Amazon S3
7 80 (11.89%) Azure Data Factory
8 54 (8.02%) Azure Synapse Analytics
9 49 (7.28%) AWS Lambda
10 46 (6.84%) Azure Cosmos DB
11 44 (6.54%) Amazon EMR
12 42 (6.24%) Azure Functions
13 39 (5.79%) GitHub
14 33 (4.90%) Snowflake
15 29 (4.31%) Amazon EC2
16 22 (3.27%) AWS Data Pipeline
17 19 (2.82%) Amazon Kinesis
18 18 (2.67%) AWS CloudFormation
18 18 (2.67%) Azure Key Vault
19 17 (2.53%) Cloud Computing
Communications & Networking
1 5 (0.74%) Cisco ASA
2 3 (0.45%) HTTP
3 2 (0.30%) Broadband
3 2 (0.30%) DNS
3 2 (0.30%) DSL
3 2 (0.30%) Internet
3 2 (0.30%) Kerberos
3 2 (0.30%) SMTP
4 1 (0.15%) DHCP
4 1 (0.15%) DMARC
4 1 (0.15%) Firewall
4 1 (0.15%) HTTPS
4 1 (0.15%) IPv4
4 1 (0.15%) IPv6
4 1 (0.15%) Remote Desktop
4 1 (0.15%) SMS
4 1 (0.15%) SNMP
4 1 (0.15%) SSH
4 1 (0.15%) SSL
4 1 (0.15%) TCP/IP
Database & Business Intelligence
1 197 (29.27%) Data Warehouse
2 185 (27.49%) Big Data
3 144 (21.40%) Data Lake
4 134 (19.91%) Hadoop
5 113 (16.79%) NoSQL
6 95 (14.12%) Power BI
7 90 (13.37%) Relational Database
8 74 (11.00%) BigQuery
8 74 (11.00%) Tableau
9 62 (9.21%) SQL Server
10 60 (8.92%) Apache Cassandra
10 60 (8.92%) Metadata
11 59 (8.77%) Apache Hive
12 58 (8.62%) HBase
13 57 (8.47%) Amazon Redshift
14 53 (7.88%) MongoDB
15 43 (6.39%) MySQL
15 43 (6.39%) PostgreSQL
16 37 (5.50%) Amazon Athena
17 33 (4.90%) Flink
Development Applications
1 109 (16.20%) Git (software)
2 94 (13.97%) Jenkins
3 32 (4.75%) JIRA
4 17 (2.53%) GitLab
5 13 (1.93%) Maven
6 10 (1.49%) Jasmine
6 10 (1.49%) JUnit
6 10 (1.49%) Oracle APEX
6 10 (1.49%) Visual Studio Team System
7 9 (1.34%) TeamCity
7 9 (1.34%) Visual Studio
8 8 (1.19%) Cucumber
8 8 (1.19%) Karma Test Runner
9 7 (1.04%) Bitbucket
10 5 (0.74%) GoCD
10 5 (0.74%) kappa
10 5 (0.74%) ProC
10 5 (0.74%) Subversion
11 4 (0.59%) CircleCI
11 4 (0.59%) IBM UrbanCode
General
1 134 (19.91%) Finance
2 117 (17.38%) Social Skills
3 96 (14.26%) Analytical Skills
4 30 (4.46%) Banking
5 26 (3.86%) Public Sector
6 24 (3.57%) Presentation Skills
7 20 (2.97%) Marketing
8 12 (1.78%) Retail
9 11 (1.63%) Aerospace
10 10 (1.49%) Publishing
11 9 (1.34%) Pharmaceutical
12 8 (1.19%) Law
12 8 (1.19%) Telecoms
13 7 (1.04%) Investment Banking
14 6 (0.89%) Games
14 6 (0.89%) Legal
14 6 (0.89%) Manufacturing
15 5 (0.74%) Organisational Skills
15 5 (0.74%) Retail Banking
16 4 (0.59%) Advertising
Job Titles
1 386 (57.36%) Data Engineer
2 88 (13.08%) Azure Engineer
3 62 (9.21%) Developer
4 42 (6.24%) Senior Data Engineer
5 37 (5.50%) Architect
6 32 (4.75%) Data Architect
7 31 (4.61%) AWS Engineer
8 30 (4.46%) Python Engineer
9 28 (4.16%) Data Scientist
10 25 (3.71%) Cloud Engineer
11 22 (3.27%) Analyst
12 21 (3.12%) DevOps Engineer
13 18 (2.67%) Big Data Engineer
14 17 (2.53%) Data Analyst
14 17 (2.53%) Senior Data Scientist
15 15 (2.23%) Python Developer
16 14 (2.08%) Machine Learning Engineer
16 14 (2.08%) Software Engineer
17 11 (1.63%) Business Intelligence Developer
17 11 (1.63%) Solutions Architect
Libraries, Frameworks & Software Standards
1 171 (25.41%) Kafka
2 72 (10.70%) PySpark
3 57 (8.47%) Spark Streaming
4 32 (4.75%) Node.js
4 32 (4.75%) Pandas
5 30 (4.46%) .NET
6 26 (3.86%) AngularJS
6 26 (3.86%) ARM Templates
7 24 (3.57%) JSON
8 23 (3.42%) Web Services
9 22 (3.27%) React
9 22 (3.27%) Spring
10 21 (3.12%) RESTful
11 18 (2.67%) REST
12 16 (2.38%) Velocity
13 11 (1.63%) .NET Framework
14 10 (1.49%) ASP.NET
14 10 (1.49%) Oracle GoldenGate
14 10 (1.49%) scikit-learn
15 9 (1.34%) pytest
Miscellaneous
1 61 (9.06%) Data Structures
2 47 (6.98%) Algorithms
3 28 (4.16%) Management Information System
4 25 (3.71%) iPad
5 17 (2.53%) Data Centre
6 16 (2.38%) Cloud Native
7 15 (2.23%) Greenfield Project
8 13 (1.93%) Public Cloud
9 9 (1.34%) Data Protection Act
9 9 (1.34%) Private Cloud
10 8 (1.19%) FMCG
10 8 (1.19%) Legacy Applications
10 8 (1.19%) Social Media
11 7 (1.04%) Hedge funds
11 7 (1.04%) Virtual Team
12 6 (0.89%) Legacy Systems
12 6 (0.89%) NHS
13 5 (0.74%) Enterprise Software
13 5 (0.74%) Replication
14 4 (0.59%) Mobile App
Operating Systems
1 27 (4.01%) Linux
2 18 (2.67%) Unix
3 9 (1.34%) Android
4 7 (1.04%) Windows
5 4 (0.59%) VMS
6 2 (0.30%) Apple iOS
6 2 (0.30%) CentOS
6 2 (0.30%) Ubuntu
7 1 (0.15%) Windows Server
Processes & Methodologies
1 265 (39.38%) Analytics
2 212 (31.50%) ETL
3 194 (28.83%) Data Engineering
4 192 (28.53%) Agile Software Development
5 171 (25.41%) DevOps
6 157 (23.33%) Data Science
7 155 (23.03%) Data Modelling
8 147 (21.84%) Business Intelligence
9 121 (17.98%) Machine Learning
10 105 (15.60%) Visualisation
11 84 (12.48%) Computer Science
12 75 (11.14%) Data Transformation
13 70 (10.40%) Scrum
14 67 (9.96%) Data Ingestion
15 63 (9.36%) Software Engineering
15 63 (9.36%) Workflow
16 61 (9.06%) Problem-Solving
17 59 (8.77%) Data Analysis
18 57 (8.47%) Statistics
19 55 (8.17%) Data Architecture
Programming Languages
1 402 (59.73%) Python
2 354 (52.60%) SQL
3 168 (24.96%) Java
4 144 (21.40%) Scala
5 35 (5.20%) JavaScript
6 34 (5.05%) R
7 32 (4.75%) C#
8 19 (2.82%) C++
8 19 (2.82%) TypeScript
9 16 (2.38%) PL/SQL
9 16 (2.38%) T-SQL
10 13 (1.93%) PowerShell
11 12 (1.78%) Data Analysis Expressions (DAX)
12 11 (1.63%) CQL
12 11 (1.63%) Ruby
13 10 (1.49%) Bash Shell
13 10 (1.49%) Go
14 9 (1.34%) Perl
15 8 (1.19%) Shell Script
16 7 (1.04%) VBA
Qualifications
1 87 (12.93%) Degree
2 45 (6.69%) Computer Science Degree
3 40 (5.94%) Security Cleared
4 34 (5.05%) SC Cleared
5 22 (3.27%) Master's Degree
6 14 (2.08%) AWS Certification
7 11 (1.63%) BPSS Clearance
8 5 (0.74%) DV Cleared
9 2 (0.30%) (ISC)2 CCSP
9 2 (0.30%) Cisco Certification
9 2 (0.30%) CISM
9 2 (0.30%) CISSP
9 2 (0.30%) Numerate Degree
9 2 (0.30%) TOGAF Certification
10 1 (0.15%) PhD
Quality Assurance & Compliance
1 68 (10.10%) Data Quality
2 29 (4.31%) QA
3 13 (1.93%) GDPR
4 3 (0.45%) Accessibility
4 3 (0.45%) COBIT
5 2 (0.30%) Disclosure Scotland
5 2 (0.30%) ISO/IEC 27001
5 2 (0.30%) NIST
6 1 (0.15%) IFRS
6 1 (0.15%) PMO
6 1 (0.15%) SLA
System Software
1 67 (9.96%) Docker
2 16 (2.38%) Active Directory
3 7 (1.04%) Apache ZooKeeper
3 7 (1.04%) HDFS
4 6 (0.89%) Apache Flume
5 3 (0.45%) Virtual Machines
Systems Management
1 102 (15.16%) Kubernetes
2 59 (8.77%) Terraform
3 22 (3.27%) Ansible
4 12 (1.78%) Opscode Chef
4 12 (1.78%) Puppet
5 8 (1.19%) Oozie
6 7 (1.04%) Hadoop YARN
6 7 (1.04%) Kibana
7 5 (0.74%) Active Directory Federation Services
7 5 (0.74%) FortiGate
8 4 (0.59%) Control-M
8 4 (0.59%) logstash
9 3 (0.45%) Grafana
9 3 (0.45%) Kubeflow
10 2 (0.30%) Istio
10 2 (0.30%) Kustomize
11 1 (0.15%) Prometheus
11 1 (0.15%) Salt
Vendors
1 141 (20.95%) Databricks
2 93 (13.82%) Google
3 92 (13.67%) Microsoft
4 38 (5.65%) Oracle
5 32 (4.75%) Cloudera
6 19 (2.82%) Informatica
7 18 (2.67%) Alteryx
8 14 (2.08%) SAP
9 12 (1.78%) Talend
10 10 (1.49%) Atlassian
11 9 (1.34%) Confluent
11 9 (1.34%) Teradata
11 9 (1.34%) Twitter
12 8 (1.19%) IBM
13 7 (1.04%) Dataiku
14 6 (0.89%) Hortonworks
15 5 (0.74%) CheckPoint
15 5 (0.74%) Cisco
16 4 (0.59%) Adobe
17 3 (0.45%) MicroStrategy