Period
to 4 December 2021

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 4 December 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Data Pipelines
England > London
6 months to
4 Dec 2021
Same period 2020 Same period 2019
Rank 218 216 263
Rank change year-on-year -2 +47 +235
Contract jobs citing Data Pipeline 492 185 344
As % of all contract jobs advertised in London 1.30% 1.14% 0.98%
As % of the Processes & Methodologies category 1.43% 1.28% 1.11%
Number of daily rates quoted 319 112 289
10th Percentile £424 £400 £425
25th Percentile £500 £463 £513
Median daily rate (50th Percentile) £575 £550 £550
Median % change year-on-year +4.55% - -
75th Percentile £675 £635 £638
90th Percentile £750 £675 £715
England median daily rate £550 £550 £550
Number of hourly rates quoted 4 4 0
Median hourly rate £46.25 £47.21 -
Median % change year-on-year -2.04% - -
England median hourly rate £26.39 £46.93 -
% change year-on-year -43.77% - -

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for process or methodology skills.

All Process and Methodology Skills
London
Contract vacancies with a requirement for process or methodology skills 34,406 14,420 31,061
As % of all contract IT jobs advertised in London 91.23% 88.77% 88.59%
Number of daily rates quoted 24,990 10,421 22,671
10th Percentile £375 £319 £325
25th Percentile £463 £413 £413
Median daily rate (50th Percentile) £550 £500 £513
Median % change year-on-year +10.00% -2.44% +2.50%
75th Percentile £663 £613 £600
90th Percentile £750 £700 £688
England median daily rate £525 £475 £475
% change year-on-year +10.53% - -
Number of hourly rates quoted 318 133 372
10th Percentile £17.18 £16.15 £12.44
25th Percentile £21.50 £22.50 £16.08
Median hourly rate £37.50 £32.00 £22.50
Median % change year-on-year +17.19% +42.22% +7.14%
75th Percentile £54.22 £45.75 £44.06
90th Percentile £70.00 £58.60 £53.75
England median hourly rate £24.08 £27.00 £22.00
% change year-on-year -10.81% +22.73% +9.15%

Data Pipeline
Job Vacancy Trend in London

Job postings citing Data Pipeline as a proportion of all IT jobs advertised in London.

Job vacancy trend for Data Pipeline in London

Data Pipeline
Contractor Daily Rate Trend in London

3-month moving average daily rate quoted in jobs citing Data Pipeline in London.

Daily rate trend for Data Pipeline in London

Data Pipeline
Daily Rate Histogram in London

Daily rate distribution for jobs citing Data Pipeline in London over the 6 months to 4 December 2021.

Daily rate histogram for Data Pipeline in London

Data Pipeline
Contractor Hourly Rate Trend in London

3-month moving average hourly rates quoted in jobs citing Data Pipeline in London.

Hourly rate trend for Data Pipeline in London

Data Pipeline
Hourly Rate Histogram in London

Hourly rate distribution of jobs citing Data Pipeline in London over the 6 months to 4 December 2021.

Hourly rate histogram for Data Pipeline in London

Data Pipeline
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the London region over the 6 months to 4 December 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -24 51 £575 +4.55% 32
South London -2 10 £613 +22.50%
West London -25 5 £570 -9.52% 10
East London - 1 £450 - 12
Data Pipeline
England

For the 6 months to 4 December 2021, IT contractor jobs citing Data Pipeline also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Data Pipeline.

1 305 (61.99%) Python
2 262 (53.25%) SQL
3 205 (41.67%) AWS
4 186 (37.80%) Apache Spark
5 182 (36.99%) Azure
6 180 (36.59%) Analytics
7 147 (29.88%) ETL
8 145 (29.47%) Kafka
9 135 (27.44%) Big Data
9 135 (27.44%) Agile Software Development
9 135 (27.44%) Data Engineering
10 124 (25.20%) Data Warehouse
11 114 (23.17%) DevOps
12 113 (22.97%) Data Modelling
13 111 (22.56%) Scala
14 110 (22.36%) Data Science
15 107 (21.75%) Java
16 105 (21.34%) Databricks
17 90 (18.29%) Apache Airflow
17 90 (18.29%) Business Intelligence
17 90 (18.29%) Finance
18 89 (18.09%) Data Lake
19 86 (17.48%) Hadoop
19 86 (17.48%) GCP
20 85 (17.28%) NoSQL
21 79 (16.06%) Kubernetes
22 76 (15.45%) Power Platform
22 76 (15.45%) Social Skills
23 75 (15.24%) Power BI
23 75 (15.24%) Jenkins

Data Pipeline
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 186 (37.80%) Apache Spark
2 90 (18.29%) Apache Airflow
3 58 (11.79%) Apache
4 29 (5.89%) Luigi
5 8 (1.63%) Jupyter
6 7 (1.42%) Tomcat
7 6 (1.22%) Apache Pig
7 6 (1.22%) Elasticsearch
8 5 (1.02%) CMS
9 4 (0.81%) SAS
10 3 (0.61%) Confluence
10 3 (0.61%) SharePoint
11 2 (0.41%) Apache Solr
11 2 (0.41%) Hyperledger
12 1 (0.20%) Apache Pulsar
12 1 (0.20%) OpenStack
12 1 (0.20%) Pachyderm
Applications
1 4 (0.81%) Microsoft Excel
1 4 (0.81%) Microsoft PowerPoint
2 1 (0.20%) Chatbot
2 1 (0.20%) SPSS
2 1 (0.20%) SPSS Modeler
Business Applications
1 4 (0.81%) Markit EDM
2 1 (0.20%) Dynamics AX
2 1 (0.20%) Dynamics CRM
2 1 (0.20%) SAP CO
2 1 (0.20%) SAP FI
2 1 (0.20%) SAP FI/CO
2 1 (0.20%) SAP Oil and Gas
Cloud Services
1 205 (41.67%) AWS
2 182 (36.99%) Azure
3 86 (17.48%) GCP
4 76 (15.45%) Power Platform
5 60 (12.20%) Serverless
6 58 (11.79%) Azure Data Factory
7 53 (10.77%) Amazon S3
8 39 (7.93%) Amazon EMR
9 37 (7.52%) GitHub
10 36 (7.32%) Azure Cosmos DB
11 32 (6.50%) AWS Lambda
12 31 (6.30%) Azure Functions
12 31 (6.30%) Snowflake
13 22 (4.47%) Azure Synapse Analytics
14 21 (4.27%) Amazon Kinesis
15 16 (3.25%) Amazon EC2
16 15 (3.05%) dbt
17 12 (2.44%) GitHub Actions
18 11 (2.24%) Cloud Computing
19 9 (1.83%) Virtual Private Cloud
Communications & Networking
1 5 (1.02%) Kerberos
2 4 (0.81%) IPFS
3 2 (0.41%) HTTPS
3 2 (0.41%) Remote Desktop
4 1 (0.20%) DHCP
4 1 (0.20%) DNS
4 1 (0.20%) HTTP
4 1 (0.20%) IPv4
4 1 (0.20%) IPv6
4 1 (0.20%) Network Security
4 1 (0.20%) SMTP
4 1 (0.20%) SNMP
4 1 (0.20%) SSH
4 1 (0.20%) TCP/IP
Database & Business Intelligence
1 135 (27.44%) Big Data
2 124 (25.20%) Data Warehouse
3 89 (18.09%) Data Lake
4 86 (17.48%) Hadoop
5 85 (17.28%) NoSQL
6 75 (15.24%) Power BI
7 56 (11.38%) Apache Cassandra
8 55 (11.18%) Tableau
9 51 (10.37%) Relational Database
10 44 (8.94%) Amazon Redshift
10 44 (8.94%) PostgreSQL
11 41 (8.33%) BigQuery
12 39 (7.93%) SQL Server
13 37 (7.52%) HBase
14 32 (6.50%) MongoDB
15 26 (5.28%) Apache Hive
15 26 (5.28%) Spark SQL
16 25 (5.08%) Flink
16 25 (5.08%) MySQL
17 23 (4.67%) Azkaban
Development Applications
1 75 (15.24%) Jenkins
2 69 (14.02%) Git (software)
3 35 (7.11%) TeamCity
4 27 (5.49%) Maven
5 25 (5.08%) JIRA
6 22 (4.47%) Gradle
7 13 (2.64%) GitLab
8 7 (1.42%) GoCD
9 6 (1.22%) CircleCI
10 5 (1.02%) Bitbucket
10 5 (1.02%) IBM UrbanCode
10 5 (1.02%) Jasmine
10 5 (1.02%) Karma Test Runner
11 4 (0.81%) Artifactory
11 4 (0.81%) Atlassian Bamboo
11 4 (0.81%) Subversion
11 4 (0.81%) Travis CI
12 3 (0.61%) RStudio
13 2 (0.41%) Robot Framework
13 2 (0.41%) Sonatype Nexus
General
1 90 (18.29%) Finance
2 76 (15.45%) Social Skills
3 61 (12.40%) Banking
4 59 (11.99%) Analytical Skills
5 34 (6.91%) Investment Banking
6 20 (4.07%) Public Sector
7 19 (3.86%) Marketing
8 18 (3.66%) Presentation Skills
9 11 (2.24%) Aerospace
10 9 (1.83%) Publishing
11 5 (1.02%) Games
11 5 (1.02%) Law
11 5 (1.02%) Legal
12 4 (0.81%) Advertising
12 4 (0.81%) Electronics
12 4 (0.81%) Manufacturing
12 4 (0.81%) Pharmaceutical
12 4 (0.81%) Retail
12 4 (0.81%) Telecoms
13 3 (0.61%) Influencing Skills
Job Titles
1 294 (59.76%) Data Engineer
2 45 (9.15%) Azure Engineer
2 45 (9.15%) Developer
3 34 (6.91%) Senior Data Engineer
4 30 (6.10%) Architect
5 29 (5.89%) AWS Engineer
6 27 (5.49%) Python Engineer
7 24 (4.88%) Cloud Engineer
8 20 (4.07%) Data Architect
9 18 (3.66%) Big Data Engineer
10 16 (3.25%) Solutions Architect
11 15 (3.05%) DevOps Engineer
12 13 (2.64%) Machine Learning Engineer
12 13 (2.64%) Python Developer
13 10 (2.03%) Analyst
13 10 (2.03%) Data Scientist
13 10 (2.03%) Software Engineer
14 9 (1.83%) Data Analyst
15 8 (1.63%) Consultant
16 7 (1.42%) Senior Developer
Libraries, Frameworks & Software Standards
1 145 (29.47%) Kafka
2 37 (7.52%) PySpark
3 27 (5.49%) Node.js
3 27 (5.49%) Spark Streaming
4 21 (4.27%) Spring
5 19 (3.86%) ARM Templates
6 18 (3.66%) Pandas
7 17 (3.46%) React
8 15 (3.05%) AngularJS
9 14 (2.85%) Web Services
10 11 (2.24%) .NET
10 11 (2.24%) REST
11 10 (2.03%) RESTful
11 10 (2.03%) Velocity
12 9 (1.83%) Akka
12 9 (1.83%) Apache Beam
12 9 (1.83%) JSON
13 7 (1.42%) Apache Avro
13 7 (1.42%) pytest
13 7 (1.42%) XML
Miscellaneous
1 32 (6.50%) Algorithms
1 32 (6.50%) Data Structures
2 22 (4.47%) Management Information System
3 13 (2.64%) Cloud Native
4 12 (2.44%) Data Centre
4 12 (2.44%) Greenfield Project
5 9 (1.83%) Public Cloud
6 8 (1.63%) Enterprise Software
6 8 (1.63%) iPad
7 7 (1.42%) Legacy Applications
7 7 (1.42%) Private Cloud
8 5 (1.02%) Data Protection Act
8 5 (1.02%) NHS
8 5 (1.02%) Renewable Energy
9 4 (0.81%) FMCG
9 4 (0.81%) Legacy Systems
9 4 (0.81%) Social Media
10 3 (0.61%) Virtual Team
11 2 (0.41%) Linux Command Line
11 2 (0.41%) OTC Derivatives
Operating Systems
1 19 (3.86%) Unix
2 17 (3.46%) Linux
3 3 (0.61%) VMS
4 2 (0.41%) Android
4 2 (0.41%) Apple iOS
5 1 (0.20%) Windows
Processes & Methodologies
1 180 (36.59%) Analytics
2 147 (29.88%) ETL
3 135 (27.44%) Agile Software Development
3 135 (27.44%) Data Engineering
4 114 (23.17%) DevOps
5 113 (22.97%) Data Modelling
6 110 (22.36%) Data Science
7 90 (18.29%) Business Intelligence
8 68 (13.82%) Machine Learning
9 61 (12.40%) Visualisation
10 55 (11.18%) Computer Science
11 50 (10.16%) Scrum
11 50 (10.16%) Workflow
12 45 (9.15%) Software Engineering
13 43 (8.74%) Data Architecture
13 43 (8.74%) Data Transformation
14 41 (8.33%) Problem-Solving
15 37 (7.52%) Microservices
15 37 (7.52%) Open Source
15 37 (7.52%) Test Automation
Programming Languages
1 305 (61.99%) Python
2 262 (53.25%) SQL
3 111 (22.56%) Scala
4 107 (21.75%) Java
5 24 (4.88%) R
6 22 (4.47%) JavaScript
7 13 (2.64%) Bash Shell
8 12 (2.44%) C#
8 12 (2.44%) Ruby
9 10 (2.03%) CQL
10 9 (1.83%) Perl
10 9 (1.83%) PowerShell
10 9 (1.83%) TypeScript
11 7 (1.42%) T-SQL
12 6 (1.22%) Data Analysis Expressions (DAX)
12 6 (1.22%) PL/SQL
13 5 (1.02%) Go
13 5 (1.02%) Kotlin
13 5 (1.02%) Shell Script
14 4 (0.81%) C++
Qualifications
1 55 (11.18%) Degree
2 35 (7.11%) Computer Science Degree
3 23 (4.67%) Security Cleared
4 20 (4.07%) SC Cleared
5 16 (3.25%) AWS Certification
6 11 (2.24%) Master's Degree
7 10 (2.03%) BPSS Clearance
8 4 (0.81%) ITIL Certification
9 2 (0.41%) PMI Certification
10 1 (0.20%) CBAP
10 1 (0.20%) DV Cleared
10 1 (0.20%) FFIEC
10 1 (0.20%) Microsoft Certification
10 1 (0.20%) PhD
10 1 (0.20%) PMP
Quality Assurance & Compliance
1 54 (10.98%) Data Quality
2 29 (5.89%) QA
3 5 (1.02%) GDPR
4 3 (0.61%) NIST
5 2 (0.41%) Accessibility
5 2 (0.41%) COBIT
5 2 (0.41%) ISO/IEC 27001
6 1 (0.20%) GLBA
6 1 (0.20%) IFRS
6 1 (0.20%) PCI DSS
System Software
1 68 (13.82%) Docker
2 8 (1.63%) Apache ZooKeeper
3 3 (0.61%) Active Directory
3 3 (0.61%) HDFS
4 1 (0.20%) Apache Flume
Systems Management
1 79 (16.06%) Kubernetes
2 52 (10.57%) Terraform
3 12 (2.44%) Ansible
4 6 (1.22%) Hadoop YARN
5 5 (1.02%) Control-M
5 5 (1.02%) Kubeflow
6 4 (0.81%) Kibana
6 4 (0.81%) Puppet
7 3 (0.61%) Grafana
8 2 (0.41%) Opscode Chef
9 1 (0.20%) Istio
9 1 (0.20%) logstash
9 1 (0.20%) Prometheus
Vendors
1 105 (21.34%) Databricks
2 47 (9.55%) Google
3 44 (8.94%) Microsoft
4 20 (4.07%) Oracle
5 19 (3.86%) Informatica
6 13 (2.64%) Cloudera
6 13 (2.64%) IBM
7 12 (2.44%) Alteryx
8 11 (2.24%) SAP
9 10 (2.03%) Twitter
10 6 (1.22%) StreamSets
10 6 (1.22%) Talend
11 4 (0.81%) Anaplan
11 4 (0.81%) Hortonworks
12 3 (0.61%) Dataiku
12 3 (0.61%) Splunk
13 2 (0.41%) Confluent
13 2 (0.41%) Palantir
13 2 (0.41%) Sybase
13 2 (0.41%) Virgin Media