Data Pipeline Contracts

Data Pipelines
UK

The following table provides summary statistics for contract job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 3 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
3 May 2024
Same period 2023 Same period 2022
Rank 197 291 261
Rank change year-on-year +94 -30 +5
Contract jobs citing Data Pipeline 533 553 965
As % of all contract jobs advertised in the UK 1.23% 0.93% 1.09%
As % of the Processes & Methodologies category 1.42% 1.04% 1.20%
Number of daily rates quoted 398 367 627
10th Percentile £390 £419 £450
25th Percentile £450 £488 £506
Median daily rate (50th Percentile) £525 £550 £600
Median % change year-on-year -4.55% -8.33% +9.09%
75th Percentile £638 £650 £713
90th Percentile £738 £738 £813
UK excluding London median daily rate £500 £525 £548
% change year-on-year -4.76% -4.24% +9.65%
Number of hourly rates quoted 8 6 8
10th Percentile £25.94 £50.63 £23.19
25th Percentile £42.00 £65.00 £26.10
Median hourly rate £62.50 £70.25 £30.13
Median % change year-on-year -11.03% +133.20% -34.51%
75th Percentile £66.38 £80.81 £47.25
90th Percentile £67.35 £86.25 £82.40
UK excluding London median hourly rate £62.50 £55.88 £20.00
% change year-on-year +11.86% +179.38% -

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for process or methodology skills.

Contract vacancies with a requirement for process or methodology skills 37,452 53,231 80,444
As % of all contract IT jobs advertised in the UK 86.33% 89.85% 90.62%
Number of daily rates quoted 24,130 36,930 56,488
10th Percentile £300 £325 £341
25th Percentile £413 £438 £425
Median daily rate (50th Percentile) £525 £550 £530
Median % change year-on-year -4.55% +3.77% +8.72%
75th Percentile £638 £650 £638
90th Percentile £750 £750 £738
UK excluding London median daily rate £500 £500 £475
% change year-on-year - +5.26% +8.57%
Number of hourly rates quoted 2,454 1,752 1,920
10th Percentile £12.75 £11.00 £12.50
25th Percentile £16.00 £16.25 £15.50
Median hourly rate £35.95 £36.00 £25.00
Median % change year-on-year -0.14% +44.00% +2.04%
75th Percentile £60.00 £65.00 £49.25
90th Percentile £72.50 £75.00 £64.53
UK excluding London median hourly rate £37.50 £35.00 £20.00
% change year-on-year +7.14% +75.00% -6.98%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Data Pipeline.

Daily rate trend for Data Pipeline in the UK

Data Pipeline
Daily Rate Histogram

Daily rate distribution for jobs citing Data Pipeline over the 6 months to 3 May 2024.

Daily rate histogram for Data Pipeline in the UK

Data Pipeline
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Data Pipeline.

Hourly rate trend for Data Pipeline in the UK

Data Pipeline
Hourly Rate Histogram

Hourly rate distribution of jobs citing Data Pipeline over the 6 months to 3 May 2024.

Hourly rate histogram for Data Pipeline in the UK

Data Pipeline
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 3 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +99 473 £550 - 314
London +66 312 £575 +2.00% 188
Work from Home +52 168 £525 -4.55% 169
UK excluding London +55 154 £500 -4.76% 137
North of England +36 52 £500 +2.56% 32
Midlands +52 43 £550 +0.92% 26
West Midlands +41 36 £550 +0.92% 18
North West +8 28 £500 +5.26% 14
Yorkshire +37 24 £563 -13.46% 15
South East -16 18 £650 +23.81% 21
East of England +28 14 £513 -2.38% 18
Scotland +33 12 £453 +6.59% 10
South West +11 10 £550 -18.52% 19
East Midlands - 7 £413 - 8
Northern Ireland - 4 £494 - 4
Wales +18 1 £500 - 8

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 86 (16.14%) Apache Spark
2 65 (12.20%) Apache
3 60 (11.26%) Apache Airflow
4 5 (0.94%) Confluence
4 5 (0.94%) Luigi
5 1 (0.19%) ArcGIS
5 1 (0.19%) Jupyter
Applications
1 8 (1.50%) Microsoft Excel
2 3 (0.56%) Microsoft Office
2 3 (0.56%) Microsoft PowerPoint
2 3 (0.56%) Microsoft Project
2 3 (0.56%) MS Visio
3 1 (0.19%) Sketch
Business Applications
1 2 (0.38%) Dynamics CRM
Cloud Services
1 232 (43.53%) AWS
2 218 (40.90%) Azure
3 80 (15.01%) Serverless
4 71 (13.32%) Azure Data Factory
5 68 (12.76%) AWS Lambda
6 67 (12.57%) GCP
7 58 (10.88%) Amazon S3
8 47 (8.82%) Snowflake
9 40 (7.50%) Power Platform
10 38 (7.13%) Azure Synapse Analytics
11 37 (6.94%) Amazon Kinesis
12 32 (6.00%) GitHub
13 31 (5.82%) Amazon EMR
13 31 (5.82%) dbt
14 29 (5.44%) AWS Step Functions
15 28 (5.25%) Azure DevOps
16 22 (4.13%) Amazon EC2
17 13 (2.44%) Amazon MSK
18 12 (2.25%) Amazon CloudWatch
18 12 (2.25%) Azure Logic Apps
Communications & Networking
1 14 (2.63%) Intrusion Detection
1 14 (2.63%) Network Security
2 5 (0.94%) FTP
3 2 (0.38%) SAN
4 1 (0.19%) Firewall
4 1 (0.19%) HTTP
4 1 (0.19%) SSL
Database & Business Intelligence
1 130 (24.39%) Data Warehouse
2 119 (22.33%) Big Data
3 95 (17.82%) Data Lake
4 63 (11.82%) Amazon Redshift
5 47 (8.82%) NoSQL
6 39 (7.32%) Power BI
7 35 (6.57%) SQL Server
8 32 (6.00%) Relational Database
9 30 (5.63%) BigQuery
10 28 (5.25%) Flink
11 24 (4.50%) Parquet
12 22 (4.13%) Tableau
13 21 (3.94%) Amazon Athena
13 21 (3.94%) DynamoDB
14 19 (3.56%) Azure SQL Database
14 19 (3.56%) Hadoop
15 18 (3.38%) PostgreSQL
16 16 (3.00%) Amazon RDS
16 16 (3.00%) Apache Hive
17 12 (2.25%) SQL Server Integration Services
Development Applications
1 60 (11.26%) Git
2 26 (4.88%) JIRA
3 9 (1.69%) Jenkins
4 7 (1.31%) GitLab
4 7 (1.31%) Mercurial
4 7 (1.31%) MLflow
5 4 (0.75%) Deequ
5 4 (0.75%) ERwin
5 4 (0.75%) JMeter
5 4 (0.75%) TestRail
6 3 (0.56%) Bitbucket
6 3 (0.56%) Poetry
6 3 (0.56%) Postman
7 2 (0.38%) ER Studio
7 2 (0.38%) Maven
7 2 (0.38%) Travis CI
8 1 (0.19%) GoCD
8 1 (0.19%) kappa
8 1 (0.19%) Selenium
8 1 (0.19%) webpack
General
1 119 (22.33%) Social Skills
2 87 (16.32%) Analytical Skills
3 71 (13.32%) Finance
4 34 (6.38%) Public Sector
5 29 (5.44%) Banking
6 15 (2.81%) Front Office
6 15 (2.81%) Presentation Skills
7 11 (2.06%) Investment Banking
7 11 (2.06%) Pharmaceutical
8 7 (1.31%) Manufacturing
8 7 (1.31%) Multimedia
8 7 (1.31%) Retail
9 5 (0.94%) Inclusion and Diversity
10 4 (0.75%) Marketing
11 3 (0.56%) Back Office
11 3 (0.56%) Games
11 3 (0.56%) Law
12 2 (0.38%) Czech Language
12 2 (0.38%) Health Technology
12 2 (0.38%) Police
Job Titles
1 281 (52.72%) Data Engineer
2 74 (13.88%) Senior
3 62 (11.63%) Developer
4 60 (11.26%) Azure Engineer
5 54 (10.13%) Azure Data Engineer
6 51 (9.57%) AWS Engineer
7 43 (8.07%) Architect
8 39 (7.32%) AWS Data Engineer
8 39 (7.32%) Senior Data Engineer
9 30 (5.63%) Python Developer
10 29 (5.44%) Lead
11 22 (4.13%) Analyst
12 21 (3.94%) Data Scientist
13 19 (3.56%) Lead Data Engineer
13 19 (3.56%) Python Engineer
14 18 (3.38%) Data Architect
15 15 (2.81%) Cloud Engineer
16 14 (2.63%) Python Data Engineer
17 13 (2.44%) Platform Engineer
17 13 (2.44%) Senior Developer
Libraries, Frameworks & Software Standards
1 66 (12.38%) PySpark
2 65 (12.20%) Kafka
3 44 (8.26%) Pandas
4 30 (5.63%) NumPy
5 26 (4.88%) JSON
6 22 (4.13%) SciPy
7 20 (3.75%) Apache Avro
8 15 (2.81%) Apache NiFi
9 12 (2.25%) REST
9 12 (2.25%) scikit-learn
10 10 (1.88%) TensorFlow
11 9 (1.69%) Elastic Stack
11 9 (1.69%) React
12 8 (1.50%) Flask
13 7 (1.31%) Great Expectations
14 6 (1.13%) ARM Templates
14 6 (1.13%) FastAPI
14 6 (1.13%) Lambda Expressions
14 6 (1.13%) Node.js
14 6 (1.13%) YAML
Miscellaneous
1 29 (5.44%) Algorithms
2 23 (4.32%) Distributed Systems
3 20 (3.75%) Derivative
4 19 (3.56%) Management Information System
5 16 (3.00%) Distributed Applications
6 15 (2.81%) Cyber Defence
6 15 (2.81%) Data Structures
7 14 (2.63%) Linux Command Line
8 13 (2.44%) Greenfield Project
9 10 (1.88%) Cloud Native
10 9 (1.69%) Distributed Computing
10 9 (1.69%) Onboarding
11 8 (1.50%) Social Media
12 7 (1.31%) Self-Motivation
13 6 (1.13%) Hedge funds
14 5 (0.94%) Analytical Mindset
14 5 (0.94%) NHS
14 5 (0.94%) Renewable Energy
15 4 (0.75%) Legacy Systems
15 4 (0.75%) Life Science
Operating Systems
1 39 (7.32%) Linux
2 30 (5.63%) Windows
3 20 (3.75%) Unix
4 12 (2.25%) Android
4 12 (2.25%) Apple iOS
5 8 (1.50%) Ubuntu
6 1 (0.19%) Mac OS
6 1 (0.19%) Windows Server
Processes & Methodologies
1 204 (38.27%) ETL
2 172 (32.27%) Data Modelling
3 171 (32.08%) Analytics
4 156 (29.27%) Data Engineering
5 112 (21.01%) CI/CD
6 110 (20.64%) DevOps
7 95 (17.82%) Problem-Solving
8 91 (17.07%) Data Science
9 88 (16.51%) Agile
10 80 (15.01%) Business Intelligence
11 78 (14.63%) Data Integration
12 63 (11.82%) Machine Learning
13 59 (11.07%) Computer Science
14 55 (10.32%) Data Ingestion
15 53 (9.94%) Test Automation
16 52 (9.76%) Data Management
17 49 (9.19%) Data Governance
17 49 (9.19%) Visualisation
18 46 (8.63%) Software Engineering
19 45 (8.44%) Data Transformation
Programming Languages
1 338 (63.41%) Python
2 280 (52.53%) SQL
3 76 (14.26%) Java
4 66 (12.38%) Scala
5 24 (4.50%) C#
6 20 (3.75%) C++
7 19 (3.56%) JavaScript
8 17 (3.19%) T-SQL
9 14 (2.63%) R
10 12 (2.25%) Rust
11 10 (1.88%) Bash
12 7 (1.31%) Bicep
12 7 (1.31%) TypeScript
13 5 (0.94%) PowerShell
14 4 (0.75%) Data Analysis Expressions
14 4 (0.75%) Perl
14 4 (0.75%) Shell Script
15 3 (0.56%) MATLAB
16 2 (0.38%) Lua
16 2 (0.38%) Ruby
Qualifications
1 100 (18.76%) Security Cleared
2 78 (14.63%) SC Cleared
3 51 (9.57%) Degree
4 36 (6.75%) Computer Science Degree
5 19 (3.56%) DV Cleared
6 11 (2.06%) AWS Certification
7 9 (1.69%) Master's Degree
8 8 (1.50%) BPSS Clearance
9 7 (1.31%) Azure Certification
10 5 (0.94%) AWS Certified Solutions Architect
10 5 (0.94%) CISSP
11 3 (0.56%) Microsoft Certification
12 2 (0.38%) AWS Certified Developer
12 2 (0.38%) AWS Certified DevOps Engineer
12 2 (0.38%) PhD
13 1 (0.19%) CTC Cleared
Quality Assurance & Compliance
1 90 (16.89%) Data Quality
2 14 (2.63%) Actionable Recommendations
3 13 (2.44%) QA
4 4 (0.75%) GDPR
4 4 (0.75%) NCSC
5 3 (0.56%) Accessibility
6 2 (0.38%) PCI DSS
7 1 (0.19%) California Consumer Privacy Act
7 1 (0.19%) JSP 440
System Software
1 48 (9.01%) Docker
2 8 (1.50%) Active Directory
2 8 (1.50%) HDFS
2 8 (1.50%) iptables
3 2 (0.38%) VMware Infrastructure
4 1 (0.19%) Apache ZooKeeper
4 1 (0.19%) Virtual Machines
Systems Management
1 60 (11.26%) Terraform
2 42 (7.88%) Kubernetes
3 8 (1.50%) Kibana
3 8 (1.50%) logstash
4 4 (0.75%) Ansible
4 4 (0.75%) Argo
5 3 (0.56%) Progress Chef
6 2 (0.38%) Hadoop YARN
6 2 (0.38%) Puppet
7 1 (0.19%) Control-M
7 1 (0.19%) Kubeflow
Vendors
1 119 (22.33%) Databricks
2 33 (6.19%) Microsoft
3 25 (4.69%) Google
4 21 (3.94%) Informatica
5 17 (3.19%) Oracle
6 12 (2.25%) SAP
7 11 (2.06%) Red Hat
7 11 (2.06%) Talend
8 8 (1.50%) Palantir
9 7 (1.31%) Cloudera
10 6 (1.13%) Foundry
11 4 (0.75%) Facebook
11 4 (0.75%) IFS
11 4 (0.75%) Twitter
12 3 (0.56%) Dataiku
12 3 (0.56%) Instagram
12 3 (0.56%) Sage
12 3 (0.56%) YouTube
13 2 (0.38%) Bloomberg
13 2 (0.38%) Salesforce