Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 28 November 2023 with a comparison to the same period in the previous 2 years.

6 months to
28 Nov 2023
Same period 2022 Same period 2021
Rank 231 253 264
Rank change year-on-year +22 +11 +82
Permanent jobs citing Data Pipeline 811 1,843 1,773
As % of all permanent jobs advertised in the UK 1.49% 1.47% 1.38%
As % of the Processes & Methodologies category 1.59% 1.53% 1.45%
Number of salaries quoted 598 1,124 1,021
10th Percentile £46,250 £51,250 £46,000
25th Percentile £52,500 £59,999 £52,500
Median annual salary (50th Percentile) £70,576 £75,000 £70,000
Median % change year-on-year -5.90% +7.14% +7.69%
75th Percentile £88,750 £87,500 £86,250
90th Percentile £105,750 £100,000 £95,000
UK excluding London median annual salary £65,175 £70,000 £57,500
% change year-on-year -6.89% +21.74% +9.52%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 51,121 120,104 122,523
As % of all permanent jobs advertised in the UK 94.09% 95.72% 95.43%
Number of salaries quoted 36,956 73,203 81,032
10th Percentile £32,500 £34,313 £32,500
25th Percentile £43,750 £45,000 £42,500
Median annual salary (50th Percentile) £60,000 £60,000 £59,000
Median % change year-on-year - +1.69% +7.27%
75th Percentile £80,000 £81,250 £77,500
90th Percentile £98,750 £100,000 £95,000
UK excluding London median annual salary £52,500 £53,000 £50,000
% change year-on-year -0.94% +6.00% +4.17%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 28 November 2023.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 28 November 2023. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +36 727 £72,500 -3.33% 364
UK excluding London +56 446 £65,175 -6.89% 192
Work from Home +12 430 £75,000 - 205
London 0 309 £80,000 +1.59% 217
North of England +65 171 £69,447 -6.78% 60
North West +97 104 £69,447 -2.53% 38
South East +85 70 £72,500 +3.57% 51
Midlands +51 66 £53,500 -23.57% 26
South West +39 60 £70,576 +19.42% 14
Yorkshire +23 58 £70,576 -5.90% 9
West Midlands +36 42 £57,000 -21.38% 12
Scotland +27 32 £70,576 +17.63% 11
East Midlands +41 24 £47,500 -20.83% 14
Wales -13 24 £50,000 -12.38% 12
East of England +100 15 £62,500 -3.85% 17
North East +31 9 £52,500 -34.38% 13
Northern Ireland +39 8 £70,000 +28.44% 3

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 131 (16.15%) Apache Spark
2 86 (10.60%) Apache Airflow
3 83 (10.23%) Apache
4 9 (1.11%) Confluence
4 9 (1.11%) SAS
5 7 (0.86%) Luigi
6 5 (0.62%) Ethereum
7 4 (0.49%) SharePoint
8 2 (0.25%) Apache Solr
8 2 (0.25%) Jupyter
9 1 (0.12%) IIS
9 1 (0.12%) Mendix
9 1 (0.12%) Tornado
Applications
1 5 (0.62%) MS Visio
2 2 (0.25%) Microsoft Office
3 1 (0.12%) SPSS
Business Applications
1 5 (0.62%) Dynamics CRM
2 1 (0.12%) FastStats
2 1 (0.12%) Ripple
2 1 (0.12%) SAP S/4HANA
Cloud Services
1 334 (41.18%) Azure
2 258 (31.81%) AWS
3 206 (25.40%) GCP
4 204 (25.15%) Power Platform
5 98 (12.08%) Azure Data Factory
6 94 (11.59%) GitHub
7 81 (9.99%) Snowflake
8 72 (8.88%) Azure DevOps
9 67 (8.26%) Serverless
10 48 (5.92%) GitHub Actions
11 47 (5.80%) Vertex AI
12 46 (5.67%) Amazon S3
13 42 (5.18%) Azure Synapse Analytics
14 38 (4.69%) AWS Lambda
15 37 (4.56%) dbt
16 25 (3.08%) Boomi
17 22 (2.71%) PaaS
18 21 (2.59%) Dagster
19 15 (1.85%) AWS Step Functions
20 13 (1.60%) Azure Cosmos DB
Communications & Networking
1 9 (1.11%) Firewall
2 5 (0.62%) SMS
3 1 (0.12%) DSL
3 1 (0.12%) FTP
3 1 (0.12%) HTTP
3 1 (0.12%) Internet
3 1 (0.12%) Intranet
3 1 (0.12%) SSL
Database & Business Intelligence
1 229 (28.24%) Data Warehouse
2 202 (24.91%) Power BI
3 194 (23.92%) Data Lake
4 133 (16.40%) Big Data
5 129 (15.91%) Tableau
6 99 (12.21%) NoSQL
7 92 (11.34%) SQL Server
8 88 (10.85%) BigQuery
8 88 (10.85%) PostgreSQL
9 85 (10.48%) Relational Database
10 59 (7.27%) Looker
11 57 (7.03%) SQL Server Integration Services
12 56 (6.91%) MySQL
13 53 (6.54%) Amazon Redshift
14 51 (6.29%) Azure SQL Database
14 51 (6.29%) Bigtable
15 48 (5.92%) Hadoop
16 43 (5.30%) Metadata
17 39 (4.81%) Delta Lake
18 37 (4.56%) DynamoDB
Development Applications
1 84 (10.36%) Jenkins
2 43 (5.30%) Git
3 21 (2.59%) JIRA
4 10 (1.23%) Spinnaker
5 6 (0.74%) Android Studio
5 6 (0.74%) SQL Server Management Studio (SSMS)
5 6 (0.74%) Xcode
6 5 (0.62%) Oracle Forms
7 4 (0.49%) CircleCI
7 4 (0.49%) GitLab
7 4 (0.49%) NUnit
8 3 (0.37%) ERwin
9 2 (0.25%) Bitbucket
9 2 (0.25%) Cucumber
9 2 (0.25%) Selenium
9 2 (0.25%) TeamCity
9 2 (0.25%) Visual Studio Team System
10 1 (0.12%) Atlassian Bamboo
10 1 (0.12%) Maven
10 1 (0.12%) Subversion
General
1 238 (29.35%) Finance
2 158 (19.48%) Social Skills
3 146 (18.00%) Banking
4 141 (17.39%) Analytical Skills
5 51 (6.29%) Retail
6 37 (4.56%) Law
7 26 (3.21%) Presentation Skills
8 22 (2.71%) Inclusion and Diversity
8 22 (2.71%) Marketing
9 18 (2.22%) Public Sector
10 13 (1.60%) Legal
11 11 (1.36%) Influencing Skills
12 10 (1.23%) Health Technology
13 8 (0.99%) Tech for Good
14 7 (0.86%) Advertising
15 4 (0.49%) Documentation Skills
15 4 (0.49%) Financial Institution
15 4 (0.49%) Games
15 4 (0.49%) Investment Banking
15 4 (0.49%) Retail Banking
Job Titles
1 416 (51.29%) Data Engineer
2 164 (20.22%) Senior
3 121 (14.92%) Senior Data Engineer
4 83 (10.23%) Lead
5 63 (7.77%) Architect
6 62 (7.64%) Data Architect
7 56 (6.91%) Developer
8 40 (4.93%) Python Engineer
9 39 (4.81%) Analyst
9 39 (4.81%) Lead Data Engineer
10 36 (4.44%) Data Analyst
11 35 (4.32%) Data Scientist
12 30 (3.70%) Python Data Engineer
13 27 (3.33%) Azure Engineer
13 27 (3.33%) DevOps Engineer
14 26 (3.21%) GCP Engineer
14 26 (3.21%) Integration Specialist
15 23 (2.84%) Solutions Architect
16 22 (2.71%) Azure Data Engineer
16 22 (2.71%) Product Specialist
Libraries, Frameworks & Software Standards
1 126 (15.54%) Kafka
2 76 (9.37%) PySpark
3 49 (6.04%) Pandas
4 18 (2.22%) NumPy
5 17 (2.10%) JSON
6 16 (1.97%) Flask
7 14 (1.73%) Apache Beam
7 14 (1.73%) XGBoost
8 12 (1.48%) JRuby
9 11 (1.36%) .NET
10 10 (1.23%) Apache Avro
10 10 (1.23%) Great Expectations
10 10 (1.23%) Node.js
10 10 (1.23%) RESTful
11 8 (0.99%) Matplotlib
11 8 (0.99%) REST
12 7 (0.86%) Django
12 7 (0.86%) SQLAlchemy
13 6 (0.74%) .NET Core
13 6 (0.74%) Middleware
Miscellaneous
1 52 (6.41%) Algorithms
1 52 (6.41%) Data Structures
2 51 (6.29%) Public Cloud
3 50 (6.17%) Replication
4 30 (3.70%) Greenfield Project
5 23 (2.84%) Product Ownership
6 20 (2.47%) Cloud Native
6 20 (2.47%) Distributed Systems
7 17 (2.10%) Management Information System
8 16 (1.97%) Renewable Energy
9 14 (1.73%) Enterprise Software
9 14 (1.73%) IoT
10 13 (1.60%) Mobile App
11 10 (1.23%) Smart Meter
12 9 (1.11%) Distributed Computing
13 7 (0.86%) Analytical Mindset
13 7 (0.86%) Hedge funds
13 7 (0.86%) Reinsurance
14 6 (0.74%) Legacy Systems
14 6 (0.74%) Virtual Team
Operating Systems
1 18 (2.22%) Linux
2 7 (0.86%) Unix
3 6 (0.74%) Android
3 6 (0.74%) Apple iOS
4 1 (0.12%) Red Hat Enterprise Linux
4 1 (0.12%) VMS
4 1 (0.12%) Windows
Processes & Methodologies
1 401 (49.45%) ETL
2 314 (38.72%) Analytics
3 266 (32.80%) Data Engineering
4 260 (32.06%) Agile
4 260 (32.06%) Business Intelligence
5 231 (28.48%) Machine Learning
6 226 (27.87%) Data Modelling
7 210 (25.89%) CI/CD
8 198 (24.41%) Visualisation
9 189 (23.30%) DevOps
10 182 (22.44%) Data Science
11 144 (17.76%) Data Integration
12 121 (14.92%) Data Visualisation
13 120 (14.80%) Problem-Solving
14 116 (14.30%) Artificial Intelligence
15 110 (13.56%) Computer Science
16 99 (12.21%) Data Analysis
17 84 (10.36%) Data Analytics
18 78 (9.62%) Data Governance
19 74 (9.12%) Infrastructure as Code
Programming Languages
1 451 (55.61%) SQL
2 422 (52.03%) Python
3 112 (13.81%) Java
4 59 (7.27%) T-SQL
5 55 (6.78%) PL/SQL
6 52 (6.41%) Scala
7 49 (6.04%) C#
8 37 (4.56%) PowerShell
9 31 (3.82%) Go
10 27 (3.33%) R
11 19 (2.34%) Bash
12 17 (2.10%) C++
13 13 (1.60%) JavaScript
13 13 (1.60%) Ruby
14 12 (1.48%) Perl
15 10 (1.23%) Data Analysis Expressions
16 9 (1.11%) TypeScript
17 5 (0.62%) Shell Script
18 4 (0.49%) Bicep
18 4 (0.49%) Kusto Query Language
Qualifications
1 151 (18.62%) Degree
2 65 (8.01%) Computer Science Degree
3 58 (7.15%) Master's Degree
4 38 (4.69%) Security Cleared
5 32 (3.95%) SC Cleared
6 15 (1.85%) Azure Certification
7 12 (1.48%) Microsoft Certification
8 8 (0.99%) DV Cleared
9 6 (0.74%) MCSA
10 5 (0.62%) PhD
11 4 (0.49%) MCSE
12 2 (0.25%) AWS Certification
12 2 (0.25%) BPSS Clearance
12 2 (0.25%) HND
13 1 (0.12%) DBS Check
13 1 (0.12%) PMP
13 1 (0.12%) PRINCE2 Certification
Quality Assurance & Compliance
1 102 (12.58%) Data Quality
2 29 (3.58%) QA
3 14 (1.73%) GDPR
4 8 (0.99%) Accessibility
5 3 (0.37%) GMP
5 3 (0.37%) SLA
6 2 (0.25%) Cyber Essentials
6 2 (0.25%) Cyber Essentials PLUS
6 2 (0.25%) ISO/IEC 27001
7 1 (0.12%) California Consumer Privacy Act
7 1 (0.12%) HIPAA
System Software
1 89 (10.97%) Docker
2 4 (0.49%) Virtual Machines
3 1 (0.12%) Active Directory
3 1 (0.12%) HDFS
Systems Management
1 101 (12.45%) Kubernetes
2 96 (11.84%) Terraform
3 20 (2.47%) Ansible
4 10 (1.23%) Grafana
5 1 (0.12%) Argo
5 1 (0.12%) Kibana
5 1 (0.12%) Kubeflow
5 1 (0.12%) Oozie
5 1 (0.12%) Progress Chef
5 1 (0.12%) Puppet
Vendors
1 148 (18.25%) Databricks
2 93 (11.47%) Google
3 84 (10.36%) Microsoft
4 33 (4.07%) Oracle
5 27 (3.33%) MuleSoft
6 20 (2.47%) Confluent
7 15 (1.85%) SAP
8 13 (1.60%) Alteryx
9 12 (1.48%) IBM
10 11 (1.36%) ServiceNow
11 10 (1.23%) Splunk
12 9 (1.11%) Dynatrace
12 9 (1.11%) PagerDuty
12 9 (1.11%) Talend
13 7 (0.86%) Informatica
14 6 (0.74%) Teradata
15 4 (0.49%) OpenAI
16 3 (0.37%) Adobe
17 2 (0.25%) Cognos
17 2 (0.25%) Peregrine