Data Pipeline Contracts

Data Pipelines
UK

The following table provides summary statistics for contract job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 28 April 2024 with a comparison to the same period in the previous 2 years.

6 months to
28 Apr 2024
Same period 2023 Same period 2022
Rank 192 288 262
Rank change year-on-year +96 -26 -6
Contract jobs citing Data Pipeline 528 553 966
As % of all contract jobs advertised in the UK 1.22% 0.93% 1.09%
As % of the Processes & Methodologies category 1.41% 1.03% 1.20%
Number of daily rates quoted 397 365 633
10th Percentile £389 £419 £451
25th Percentile £450 £488 £511
Median daily rate (50th Percentile) £525 £550 £600
Median % change year-on-year -4.55% -8.33% +9.09%
75th Percentile £638 £650 £713
90th Percentile £738 £738 £800
UK excluding London median daily rate £500 £525 £550
% change year-on-year -4.76% -4.55% +10.00%
Number of hourly rates quoted 8 6 7
10th Percentile £25.94 £50.63 £22.38
25th Percentile £42.00 £65.00 £26.69
Median hourly rate £62.50 £70.25 £32.00
Median % change year-on-year -11.03% +119.53% -30.43%
75th Percentile £66.38 £80.81 £57.75
90th Percentile £67.35 £86.25 £83.61
UK excluding London median hourly rate £62.50 £55.88 £20.00
% change year-on-year +11.86% +179.38% -

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for process or methodology skills.

Contract vacancies with a requirement for process or methodology skills 37,383 53,667 80,409
As % of all contract IT jobs advertised in the UK 86.31% 89.90% 90.57%
Number of daily rates quoted 24,097 37,244 56,431
10th Percentile £300 £325 £340
25th Percentile £413 £438 £425
Median daily rate (50th Percentile) £525 £550 £525
Median % change year-on-year -4.55% +4.76% +8.25%
75th Percentile £638 £650 £638
90th Percentile £750 £750 £738
UK excluding London median daily rate £500 £500 £475
% change year-on-year - +5.26% +9.20%
Number of hourly rates quoted 2,426 1,746 1,925
10th Percentile £12.75 £11.00 £12.50
25th Percentile £16.00 £16.25 £15.25
Median hourly rate £35.00 £37.30 £25.00
Median % change year-on-year -6.17% +49.20% -
75th Percentile £59.44 £65.00 £49.25
90th Percentile £72.50 £75.00 £63.75
UK excluding London median hourly rate £36.05 £36.00 £20.00
% change year-on-year +0.13% +80.00% -8.51%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Data Pipeline.

Daily rate trend for Data Pipeline in the UK

Data Pipeline
Daily Rate Histogram

Daily rate distribution for jobs citing Data Pipeline over the 6 months to 28 April 2024.

Daily rate histogram for Data Pipeline in the UK

Data Pipeline
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Data Pipeline.

Hourly rate trend for Data Pipeline in the UK

Data Pipeline
Hourly Rate Histogram

Hourly rate distribution of jobs citing Data Pipeline over the 6 months to 28 April 2024.

Hourly rate histogram for Data Pipeline in the UK

Data Pipeline
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 28 April 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +89 467 £550 - 305
London +79 308 £575 +1.77% 181
Work from Home +49 167 £525 -4.55% 158
UK excluding London +69 152 £500 -4.76% 135
North of England +38 52 £500 +2.56% 33
Midlands +49 42 £550 +0.92% 26
West Midlands +48 37 £550 +0.92% 17
North West +14 28 £500 +5.26% 19
Yorkshire +38 24 £563 -13.46% 10
South East -13 17 £650 +20.93% 23
East of England +21 14 £513 -2.38% 13
Scotland +30 12 £453 +6.59% 12
South West +11 10 £550 -2.22% 15
East Midlands - 5 £425 - 9
Northern Ireland - 4 £494 - 5
Wales +12 1 £500 - 9

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 81 (15.34%) Apache Spark
2 64 (12.12%) Apache
3 61 (11.55%) Apache Airflow
4 6 (1.14%) Confluence
5 5 (0.95%) Luigi
6 1 (0.19%) ArcGIS
6 1 (0.19%) Jupyter
Applications
1 7 (1.33%) Microsoft Excel
2 2 (0.38%) Microsoft Office
2 2 (0.38%) Microsoft PowerPoint
2 2 (0.38%) Microsoft Project
2 2 (0.38%) MS Visio
3 1 (0.19%) Sketch
3 1 (0.19%) Spreadsheet
Business Applications
1 2 (0.38%) Dynamics CRM
Cloud Services
1 236 (44.70%) AWS
2 207 (39.20%) Azure
3 77 (14.58%) Serverless
4 67 (12.69%) AWS Lambda
5 66 (12.50%) GCP
6 63 (11.93%) Azure Data Factory
7 60 (11.36%) Amazon S3
8 47 (8.90%) Snowflake
9 37 (7.01%) Amazon Kinesis
10 36 (6.82%) Power Platform
11 32 (6.06%) Amazon EMR
11 32 (6.06%) Azure Synapse Analytics
12 31 (5.87%) GitHub
13 29 (5.49%) dbt
14 28 (5.30%) Azure DevOps
15 27 (5.11%) AWS Step Functions
16 21 (3.98%) Amazon EC2
17 13 (2.46%) Amazon MSK
18 12 (2.27%) Amazon CloudWatch
18 12 (2.27%) Cloud Computing
Communications & Networking
1 14 (2.65%) Intrusion Detection
1 14 (2.65%) Network Security
2 5 (0.95%) FTP
3 2 (0.38%) SAN
4 1 (0.19%) Firewall
4 1 (0.19%) HTTP
4 1 (0.19%) SSL
Database & Business Intelligence
1 125 (23.67%) Data Warehouse
2 115 (21.78%) Big Data
3 88 (16.67%) Data Lake
4 66 (12.50%) Amazon Redshift
5 46 (8.71%) NoSQL
6 35 (6.63%) Power BI
7 34 (6.44%) SQL Server
8 30 (5.68%) BigQuery
8 30 (5.68%) Relational Database
9 28 (5.30%) Flink
10 22 (4.17%) DynamoDB
11 20 (3.79%) Hadoop
12 19 (3.60%) Amazon Athena
12 19 (3.60%) Parquet
12 19 (3.60%) Tableau
13 18 (3.41%) PostgreSQL
14 17 (3.22%) Apache Hive
15 16 (3.03%) Amazon RDS
16 14 (2.65%) Azure SQL Database
17 12 (2.27%) Metadata
Development Applications
1 61 (11.55%) Git
2 28 (5.30%) JIRA
3 10 (1.89%) Jenkins
4 8 (1.52%) MLflow
5 7 (1.33%) GitLab
5 7 (1.33%) Mercurial
6 4 (0.76%) ERwin
7 3 (0.57%) Bitbucket
7 3 (0.57%) Deequ
7 3 (0.57%) JMeter
7 3 (0.57%) Poetry
7 3 (0.57%) Postman
7 3 (0.57%) TestRail
8 2 (0.38%) ER Studio
8 2 (0.38%) Maven
8 2 (0.38%) Travis CI
9 1 (0.19%) GoCD
9 1 (0.19%) kappa
9 1 (0.19%) webpack
General
1 116 (21.97%) Social Skills
2 88 (16.67%) Analytical Skills
3 70 (13.26%) Finance
4 34 (6.44%) Public Sector
5 28 (5.30%) Banking
6 15 (2.84%) Front Office
6 15 (2.84%) Presentation Skills
7 11 (2.08%) Investment Banking
7 11 (2.08%) Pharmaceutical
8 7 (1.33%) Manufacturing
8 7 (1.33%) Multimedia
8 7 (1.33%) Retail
9 5 (0.95%) Inclusion and Diversity
10 4 (0.76%) Marketing
11 3 (0.57%) Back Office
11 3 (0.57%) Games
11 3 (0.57%) Law
12 2 (0.38%) Czech Language
12 2 (0.38%) Influencing Skills
12 2 (0.38%) Wearables
Job Titles
1 276 (52.27%) Data Engineer
2 73 (13.83%) Senior
3 63 (11.93%) Developer
4 54 (10.23%) Azure Engineer
5 52 (9.85%) AWS Engineer
6 48 (9.09%) Azure Data Engineer
7 42 (7.95%) Architect
8 40 (7.58%) AWS Data Engineer
9 38 (7.20%) Senior Data Engineer
10 30 (5.68%) Python Developer
11 29 (5.49%) Lead
12 22 (4.17%) Analyst
12 22 (4.17%) Data Scientist
13 20 (3.79%) Python Engineer
14 19 (3.60%) Lead Data Engineer
15 18 (3.41%) Data Architect
16 15 (2.84%) Cloud Engineer
17 14 (2.65%) Python Data Engineer
17 14 (2.65%) Senior Developer
18 13 (2.46%) Data Analyst
Libraries, Frameworks & Software Standards
1 65 (12.31%) Kafka
2 59 (11.17%) PySpark
3 46 (8.71%) Pandas
4 31 (5.87%) NumPy
5 25 (4.73%) JSON
6 22 (4.17%) SciPy
7 17 (3.22%) Apache Avro
8 15 (2.84%) Apache NiFi
9 14 (2.65%) REST
10 13 (2.46%) scikit-learn
11 10 (1.89%) TensorFlow
12 9 (1.70%) Elastic Stack
12 9 (1.70%) React
13 8 (1.52%) Flask
14 6 (1.14%) .NET
14 6 (1.14%) FastAPI
14 6 (1.14%) GraphQL
14 6 (1.14%) Node.js
14 6 (1.14%) XML
14 6 (1.14%) YAML
Miscellaneous
1 31 (5.87%) Algorithms
2 24 (4.55%) Distributed Systems
3 20 (3.79%) Derivative
4 19 (3.60%) Management Information System
5 16 (3.03%) Distributed Applications
6 15 (2.84%) Cyber Defence
6 15 (2.84%) Data Structures
7 14 (2.65%) Linux Command Line
8 12 (2.27%) Greenfield Project
9 11 (2.08%) Cloud Native
9 11 (2.08%) Distributed Computing
10 10 (1.89%) Onboarding
11 8 (1.52%) Social Media
12 7 (1.33%) Self-Motivation
13 5 (0.95%) Analytical Mindset
13 5 (0.95%) NHS
13 5 (0.95%) Renewable Energy
14 4 (0.76%) Hedge funds
14 4 (0.76%) Life Science
14 4 (0.76%) Security Posture
Operating Systems
1 38 (7.20%) Linux
2 30 (5.68%) Windows
3 20 (3.79%) Unix
4 12 (2.27%) Android
4 12 (2.27%) Apple iOS
5 8 (1.52%) Ubuntu
6 1 (0.19%) Mac OS
6 1 (0.19%) Windows Server
Processes & Methodologies
1 199 (37.69%) ETL
2 172 (32.58%) Data Modelling
3 162 (30.68%) Analytics
4 151 (28.60%) Data Engineering
5 107 (20.27%) CI/CD
6 105 (19.89%) DevOps
7 95 (17.99%) Problem-Solving
8 94 (17.80%) Data Science
9 85 (16.10%) Agile
10 78 (14.77%) Data Integration
11 75 (14.20%) Business Intelligence
12 64 (12.12%) Machine Learning
13 57 (10.80%) Computer Science
14 54 (10.23%) Data Ingestion
15 51 (9.66%) Data Management
16 49 (9.28%) Test Automation
17 48 (9.09%) Data Governance
17 48 (9.09%) Visualisation
18 44 (8.33%) Software Engineering
19 42 (7.95%) Data Transformation
Programming Languages
1 334 (63.26%) Python
2 274 (51.89%) SQL
3 74 (14.02%) Java
4 62 (11.74%) Scala
5 24 (4.55%) C#
6 19 (3.60%) C++
6 19 (3.60%) JavaScript
7 16 (3.03%) R
8 14 (2.65%) T-SQL
9 12 (2.27%) Rust
10 8 (1.52%) Bash
11 7 (1.33%) Bicep
11 7 (1.33%) TypeScript
12 5 (0.95%) PowerShell
13 4 (0.76%) Data Analysis Expressions
13 4 (0.76%) Perl
13 4 (0.76%) Shell Script
14 3 (0.57%) MATLAB
14 3 (0.57%) PL/SQL
15 2 (0.38%) PHP
Qualifications
1 102 (19.32%) Security Cleared
2 80 (15.15%) SC Cleared
3 50 (9.47%) Degree
4 34 (6.44%) Computer Science Degree
5 19 (3.60%) DV Cleared
6 11 (2.08%) AWS Certification
7 9 (1.70%) Master's Degree
8 8 (1.52%) BPSS Clearance
9 7 (1.33%) Azure Certification
10 5 (0.95%) AWS Certified Solutions Architect
10 5 (0.95%) CISSP
11 3 (0.57%) Microsoft Certification
12 2 (0.38%) AWS Certified Developer
12 2 (0.38%) AWS Certified DevOps Engineer
12 2 (0.38%) PhD
13 1 (0.19%) CTC Cleared
Quality Assurance & Compliance
1 89 (16.86%) Data Quality
2 14 (2.65%) Actionable Recommendations
3 13 (2.46%) QA
4 4 (0.76%) GDPR
4 4 (0.76%) NCSC
5 3 (0.57%) Accessibility
6 2 (0.38%) PCI DSS
7 1 (0.19%) California Consumer Privacy Act
7 1 (0.19%) JSP 440
System Software
1 47 (8.90%) Docker
2 8 (1.52%) Active Directory
2 8 (1.52%) HDFS
2 8 (1.52%) iptables
3 2 (0.38%) VMware Infrastructure
4 1 (0.19%) Apache ZooKeeper
4 1 (0.19%) Virtual Machines
Systems Management
1 57 (10.80%) Terraform
2 42 (7.95%) Kubernetes
3 8 (1.52%) Kibana
3 8 (1.52%) logstash
4 4 (0.76%) Ansible
4 4 (0.76%) Argo
5 3 (0.57%) Progress Chef
6 2 (0.38%) Hadoop YARN
6 2 (0.38%) Puppet
7 1 (0.19%) Control-M
7 1 (0.19%) Kubeflow
Vendors
1 109 (20.64%) Databricks
2 32 (6.06%) Microsoft
3 25 (4.73%) Google
4 21 (3.98%) Informatica
5 19 (3.60%) Oracle
6 11 (2.08%) Red Hat
6 11 (2.08%) SAP
6 11 (2.08%) Talend
7 9 (1.70%) Cloudera
8 8 (1.52%) Palantir
9 6 (1.14%) Foundry
10 4 (0.76%) Facebook
10 4 (0.76%) IFS
10 4 (0.76%) Twitter
11 3 (0.57%) Dataiku
11 3 (0.57%) Instagram
11 3 (0.57%) YouTube
12 2 (0.38%) OpenAI
12 2 (0.38%) ServiceNow
12 2 (0.38%) Workday