Data Pipeline Contracts

Data Pipelines
UK

The following table provides summary statistics for contract job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Data Pipeline over the 6 months to 23 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
23 May 2024
Same period 2023 Same period 2022
Rank 183 285 269
Rank change year-on-year +102 -16 -8
Contract jobs citing Data Pipeline 532 512 926
As % of all contract jobs advertised in the UK 1.25% 0.91% 1.06%
As % of the Processes & Methodologies category 1.46% 1.01% 1.17%
Number of daily rates quoted 395 355 609
10th Percentile £368 £417 £450
25th Percentile £450 £494 £506
Median daily rate (50th Percentile) £525 £550 £600
Median % change year-on-year -4.55% -8.33% +9.09%
75th Percentile £644 £675 £714
90th Percentile £745 £738 £813
UK excluding London median daily rate £505 £525 £550
% change year-on-year -3.81% -4.55% +10.00%
Number of hourly rates quoted 7 5 6
10th Percentile £25.34 £47.75 £27.00
25th Percentile £37.25 £65.00 £28.56
Median hourly rate £62.50 £75.50 £33.75
Median % change year-on-year -17.22% +123.70% -17.68%
75th Percentile £66.25 £82.50 £68.25
90th Percentile £67.25 £87.00 £84.83
UK excluding London median hourly rate £62.50 £55.88 -
% change year-on-year +11.86% - -

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for process or methodology skills.

Contract vacancies with a requirement for process or methodology skills 36,537 50,741 79,365
As % of all contract IT jobs advertised in the UK 85.91% 89.71% 90.64%
Number of daily rates quoted 23,555 35,132 55,704
10th Percentile £300 £325 £345
25th Percentile £413 £439 £431
Median daily rate (50th Percentile) £525 £550 £535
Median % change year-on-year -4.55% +2.80% +8.08%
75th Percentile £638 £650 £638
90th Percentile £750 £750 £738
UK excluding London median daily rate £500 £500 £483
% change year-on-year - +3.63% +9.66%
Number of hourly rates quoted 2,471 1,680 1,899
10th Percentile £12.75 £10.63 £12.50
25th Percentile £16.00 £15.74 £15.50
Median hourly rate £36.00 £36.00 £25.68
Median % change year-on-year - +40.19% +7.00%
75th Percentile £60.00 £65.00 £49.56
90th Percentile £72.50 £75.00 £65.00
UK excluding London median hourly rate £37.75 £35.00 £20.00
% change year-on-year +7.86% +75.00% -4.99%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Data Pipeline.

Daily rate trend for Data Pipeline in the UK

Data Pipeline
Daily Rate Histogram

Daily rate distribution for jobs citing Data Pipeline over the 6 months to 23 May 2024.

Daily rate histogram for Data Pipeline in the UK

Data Pipeline
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Data Pipeline.

Hourly rate trend for Data Pipeline in the UK

Data Pipeline
Hourly Rate Histogram

Hourly rate distribution of jobs citing Data Pipeline over the 6 months to 23 May 2024.

Hourly rate histogram for Data Pipeline in the UK

Data Pipeline
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 23 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Jobs
England +93 476 £550 - 329
London +85 322 £575 +1.77% 196
Work from Home +54 168 £525 -4.55% 158
UK excluding London +55 146 £505 -3.81% 155
North of England +42 51 £500 -8.05% 43
Midlands +53 43 £550 +0.46% 26
West Midlands +30 35 £550 +0.92% 21
North West +18 27 £500 - 23
Yorkshire +37 24 £563 -13.46% 17
South East -15 19 £475 -9.52% 25
Scotland +42 12 £453 +5.96% 18
South West +6 11 £550 -18.52% 22
East Midlands +22 8 £400 -40.74% 5
East of England +18 8 £538 +2.38% 10
Wales +20 1 £500 - 4
Northern Ireland - 1 £425 - 8

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 84 (15.79%) Apache Spark
2 69 (12.97%) Apache Airflow
3 68 (12.78%) Apache
4 5 (0.94%) Confluence
5 4 (0.75%) Luigi
6 3 (0.56%) SharePoint
7 1 (0.19%) ArcGIS
7 1 (0.19%) Jupyter
Applications
1 13 (2.44%) Microsoft Excel
2 6 (1.13%) Microsoft Office
3 4 (0.75%) Microsoft PowerPoint
3 4 (0.75%) Microsoft Project
3 4 (0.75%) MS Visio
4 2 (0.38%) Sketch
Business Applications
1 2 (0.38%) Dynamics CRM
Cloud Services
1 221 (41.54%) AWS
2 211 (39.66%) Azure
3 75 (14.10%) Serverless
4 68 (12.78%) GCP
5 66 (12.41%) Azure Data Factory
6 64 (12.03%) AWS Lambda
7 61 (11.47%) Amazon S3
8 54 (10.15%) Snowflake
9 38 (7.14%) Power Platform
10 36 (6.77%) Azure Synapse Analytics
11 33 (6.20%) Amazon EMR
11 33 (6.20%) GitHub
12 32 (6.02%) dbt
13 30 (5.64%) Azure DevOps
14 27 (5.08%) Amazon Kinesis
15 26 (4.89%) AWS Step Functions
16 22 (4.14%) Amazon EC2
17 13 (2.44%) Amazon MSK
17 13 (2.44%) Azure Logic Apps
18 12 (2.26%) Cloud Computing
Communications & Networking
1 14 (2.63%) Intrusion Detection
1 14 (2.63%) Network Security
2 5 (0.94%) FTP
3 2 (0.38%) SAN
4 1 (0.19%) Firewall
Database & Business Intelligence
1 143 (26.88%) Data Warehouse
2 110 (20.68%) Big Data
3 88 (16.54%) Data Lake
4 58 (10.90%) Amazon Redshift
5 43 (8.08%) NoSQL
6 38 (7.14%) Power BI
7 36 (6.77%) BigQuery
8 35 (6.58%) Relational Database
9 32 (6.02%) SQL Server
10 26 (4.89%) Flink
11 24 (4.51%) Amazon Athena
12 23 (4.32%) Parquet
13 21 (3.95%) Tableau
14 19 (3.57%) Azure SQL Database
14 19 (3.57%) Hadoop
15 17 (3.20%) DynamoDB
16 14 (2.63%) Amazon RDS
16 14 (2.63%) PostgreSQL
17 13 (2.44%) Delta Lake
17 13 (2.44%) Spark SQL
Development Applications
1 56 (10.53%) Git
2 27 (5.08%) JIRA
3 12 (2.26%) Jenkins
4 7 (1.32%) GitLab
4 7 (1.32%) MLflow
5 5 (0.94%) JMeter
5 5 (0.94%) TestRail
6 4 (0.75%) Deequ
6 4 (0.75%) ERwin
6 4 (0.75%) Mercurial
7 3 (0.56%) Bitbucket
7 3 (0.56%) Postman
8 2 (0.38%) Bugzilla
8 2 (0.38%) ER Studio
8 2 (0.38%) JUnit
8 2 (0.38%) Selenium
8 2 (0.38%) TestNG
8 2 (0.38%) Travis CI
9 1 (0.19%) GoCD
9 1 (0.19%) kappa
General
1 121 (22.74%) Social Skills
2 88 (16.54%) Analytical Skills
3 80 (15.04%) Finance
4 35 (6.58%) Public Sector
5 28 (5.26%) Banking
6 15 (2.82%) Front Office
6 15 (2.82%) Presentation Skills
7 13 (2.44%) Pharmaceutical
8 8 (1.50%) Investment Banking
8 8 (1.50%) Marketing
9 7 (1.32%) Manufacturing
10 6 (1.13%) Retail
11 4 (0.75%) Inclusion and Diversity
11 4 (0.75%) Multimedia
12 3 (0.56%) Back Office
12 3 (0.56%) Financial Institution
12 3 (0.56%) Games
12 3 (0.56%) Law
13 2 (0.38%) Czech Language
13 2 (0.38%) Influencing Skills
Job Titles
1 275 (51.69%) Data Engineer
2 76 (14.29%) Senior
3 60 (11.28%) Developer
4 59 (11.09%) Azure Engineer
5 53 (9.96%) Azure Data Engineer
6 50 (9.40%) AWS Engineer
7 45 (8.46%) Senior Data Engineer
8 43 (8.08%) Architect
9 36 (6.77%) AWS Data Engineer
10 35 (6.58%) Lead
11 27 (5.08%) Analyst
12 23 (4.32%) Python Developer
13 22 (4.14%) Data Scientist
14 20 (3.76%) Lead Data Engineer
15 19 (3.57%) Data Analyst
15 19 (3.57%) Snowflake Engineer
16 16 (3.01%) Platform Engineer
17 15 (2.82%) Data Architect
17 15 (2.82%) Python Engineer
18 14 (2.63%) Snowflake Data Engineer
Libraries, Frameworks & Software Standards
1 72 (13.53%) PySpark
2 63 (11.84%) Kafka
3 36 (6.77%) Pandas
4 25 (4.70%) JSON
5 24 (4.51%) NumPy
6 20 (3.76%) Apache Avro
7 17 (3.20%) SciPy
8 15 (2.82%) Apache NiFi
9 11 (2.07%) scikit-learn
9 11 (2.07%) TensorFlow
10 10 (1.88%) REST
11 8 (1.50%) Elastic Stack
12 7 (1.32%) React
13 6 (1.13%) ARM Templates
13 6 (1.13%) FastAPI
13 6 (1.13%) Flask
13 6 (1.13%) Node.js
13 6 (1.13%) XML
14 5 (0.94%) .NET
14 5 (0.94%) Great Expectations
Miscellaneous
1 29 (5.45%) Algorithms
2 25 (4.70%) Distributed Systems
3 22 (4.14%) Management Information System
4 19 (3.57%) Data Structures
4 19 (3.57%) Derivative
5 15 (2.82%) Cyber Defence
5 15 (2.82%) Distributed Applications
6 12 (2.26%) Linux Command Line
7 11 (2.07%) Greenfield Project
8 10 (1.88%) Self-Motivation
9 9 (1.69%) Distributed Computing
9 9 (1.69%) Onboarding
10 7 (1.32%) Cloud Native
10 7 (1.32%) Social Media
11 6 (1.13%) Hedge funds
11 6 (1.13%) Renewable Energy
12 5 (0.94%) Analytical Mindset
12 5 (0.94%) Life Science
12 5 (0.94%) NHS
12 5 (0.94%) Security Posture
Operating Systems
1 36 (6.77%) Linux
2 27 (5.08%) Windows
3 15 (2.82%) Unix
4 12 (2.26%) Android
4 12 (2.26%) Apple iOS
5 8 (1.50%) Ubuntu
6 1 (0.19%) VMS
6 1 (0.19%) Windows Server
Processes & Methodologies
1 213 (40.04%) ETL
2 177 (33.27%) Data Modelling
3 174 (32.71%) Analytics
4 165 (31.02%) Data Engineering
5 117 (21.99%) CI/CD
6 112 (21.05%) DevOps
7 98 (18.42%) Problem-Solving
8 95 (17.86%) Agile
9 91 (17.11%) Data Science
10 79 (14.85%) Business Intelligence
11 75 (14.10%) Data Integration
12 62 (11.65%) Machine Learning
13 60 (11.28%) Computer Science
14 57 (10.71%) Data Ingestion
15 53 (9.96%) Data Management
16 50 (9.40%) Data Governance
17 49 (9.21%) Visualisation
18 47 (8.83%) Data Transformation
18 47 (8.83%) Test Automation
19 42 (7.89%) Continuous Integration
Programming Languages
1 329 (61.84%) Python
2 285 (53.57%) SQL
3 79 (14.85%) Java
4 66 (12.41%) Scala
5 22 (4.14%) C++
6 21 (3.95%) C#
7 19 (3.57%) JavaScript
8 17 (3.20%) T-SQL
9 14 (2.63%) R
10 12 (2.26%) Rust
11 10 (1.88%) Bash
12 7 (1.32%) TypeScript
13 6 (1.13%) Bicep
13 6 (1.13%) Go
14 5 (0.94%) VBA
15 4 (0.75%) Perl
15 4 (0.75%) PowerShell
15 4 (0.75%) Shell Script
16 3 (0.56%) Data Analysis Expressions
16 3 (0.56%) MATLAB
Qualifications
1 94 (17.67%) Security Cleared
2 72 (13.53%) SC Cleared
3 52 (9.77%) Degree
4 37 (6.95%) Computer Science Degree
5 17 (3.20%) DV Cleared
6 10 (1.88%) Master's Degree
7 8 (1.50%) AWS Certification
7 8 (1.50%) Azure Certification
7 8 (1.50%) BPSS Clearance
8 6 (1.13%) CISSP
9 3 (0.56%) AWS Certified Solutions Architect
9 3 (0.56%) CISA
9 3 (0.56%) CRISC
9 3 (0.56%) Microsoft Certification
10 2 (0.38%) AWS Certified Developer
10 2 (0.38%) AWS Certified DevOps Engineer
10 2 (0.38%) PhD
11 1 (0.19%) (ISC)2 CCSP
11 1 (0.19%) Cisco Certification
11 1 (0.19%) CTC Cleared
Quality Assurance & Compliance
1 93 (17.48%) Data Quality
2 14 (2.63%) Actionable Recommendations
2 14 (2.63%) QA
3 7 (1.32%) GDPR
4 5 (0.94%) Accessibility
5 3 (0.56%) NCSC
6 2 (0.38%) NIST
6 2 (0.38%) PCI DSS
7 1 (0.19%) California Consumer Privacy Act
7 1 (0.19%) ISO/IEC 27001
System Software
1 46 (8.65%) Docker
2 8 (1.50%) Active Directory
2 8 (1.50%) iptables
3 7 (1.32%) HDFS
4 2 (0.38%) VMware Infrastructure
5 1 (0.19%) Virtual Machines
Systems Management
1 55 (10.34%) Terraform
2 46 (8.65%) Kubernetes
3 8 (1.50%) Kibana
3 8 (1.50%) logstash
4 7 (1.32%) Ansible
5 5 (0.94%) Argo
6 3 (0.56%) Progress Chef
7 2 (0.38%) Puppet
8 1 (0.19%) Control-M
8 1 (0.19%) Hadoop YARN
8 1 (0.19%) Kubeflow
Vendors
1 118 (22.18%) Databricks
2 35 (6.58%) Microsoft
3 32 (6.02%) Google
4 18 (3.38%) Informatica
5 13 (2.44%) Oracle
6 10 (1.88%) Red Hat
6 10 (1.88%) SAP
7 8 (1.50%) Palantir
8 7 (1.32%) Ab Initio
9 6 (1.13%) Foundry
10 5 (0.94%) Cloudera
10 5 (0.94%) Talend
11 4 (0.75%) Facebook
11 4 (0.75%) IFS
11 4 (0.75%) Twitter
12 3 (0.56%) Alteryx
12 3 (0.56%) IBM
12 3 (0.56%) Sage
12 3 (0.56%) Splunk
12 3 (0.56%) TIBCO