Period
to 25 September 2021

The following table provides summary statistics for permanent job vacancies advertised in London with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 25 September 2021 with a comparison to the same period in the previous 2 years.

Data Pipelines
England > London
6 months to
25 Sep 2021
Same period 2020 Same period 2019
Rank 198 233 308
Rank change year-on-year +35 +75 +208
Permanent jobs citing Data Pipeline 949 280 595
As % of all permanent jobs advertised in London 1.94% 1.39% 1.07%
As % of the Processes & Methodologies category 2.03% 1.48% 1.14%
Number of salaries quoted 592 218 501
10th Percentile £47,500 £53,375 £48,750
25th Percentile £57,500 £60,000 £57,500
Median annual salary (50th Percentile) £73,000 £70,000 £70,000
Median % change year-on-year +4.29% - +3.70%
75th Percentile £87,500 £85,000 £85,000
90th Percentile £96,125 £95,000 £102,500
England median annual salary £70,000 £65,000 £70,000
% change year-on-year +7.69% -7.14% +7.69%

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies advertised in London with a requirement for process or methodology skills.

All Process and Methodology Skills
London
Permanent vacancies with a requirement for process or methodology skills 46,782 18,906 52,181
As % of all permanent IT jobs advertised in London 95.41% 94.09% 93.50%
Number of salaries quoted 30,541 15,834 42,477
10th Percentile £42,500 £42,500 £36,250
25th Percentile £52,500 £52,500 £47,500
Median annual salary (50th Percentile) £70,000 £70,000 £65,000
Median % change year-on-year - +7.69% +4.00%
75th Percentile £88,750 £87,500 £82,500
90th Percentile £105,000 £110,000 £100,000
England median annual salary £57,500 £57,500 £52,500
% change year-on-year - +9.52% +2.94%

Data Pipeline
Job Vacancy Trend in London

Job postings citing Data Pipeline as a proportion of all IT jobs advertised in London.

Job vacancy trend for Data Pipeline in London

Data Pipeline
Salary Trend in London

3-month moving average salary quoted in jobs citing Data Pipeline in London.

Salary trend for Data Pipeline in London

Data Pipeline
Salary Histogram in London

Salary distribution for jobs citing Data Pipeline in London over the 6 months to 25 September 2021.

Salary histogram for Data Pipeline in London

Data Pipeline
Job Locations in London

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the London region over the 6 months to 25 September 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London +18 135 £72,500 - 40
West London -48 29 £80,000 - 19
East London -7 15 £70,000 +3.70% 14
South London -36 8 - - 1
Data Pipeline
England

For the 6 months to 25 September 2021, IT jobs citing Data Pipeline also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads across the London region with a requirement for Data Pipeline.

1 750 (79.03%) Python
2 578 (60.91%) AWS
3 554 (58.38%) SQL
4 477 (50.26%) Analytics
5 407 (42.89%) Data Science
6 353 (37.20%) ETL
7 328 (34.56%) Apache Spark
8 311 (32.77%) Data Engineering
9 302 (31.82%) GCP
10 300 (31.61%) Agile Software Development
10 300 (31.61%) Big Data
11 267 (28.13%) Azure
12 257 (27.08%) Data Warehouse
13 250 (26.34%) Docker
14 243 (25.61%) Software Engineering
15 220 (23.18%) Data Modelling
16 216 (22.76%) Kafka
17 208 (21.92%) Social Skills
18 206 (21.71%) Java
19 204 (21.50%) Artificial Intelligence
20 203 (21.39%) Machine Learning
21 201 (21.18%) Business Intelligence
21 201 (21.18%) Kubernetes
22 184 (19.39%) NoSQL
23 183 (19.28%) DevOps
24 173 (18.23%) Hadoop
25 168 (17.70%) Snowflake
26 166 (17.49%) Google
27 164 (17.28%) Data Lake
28 162 (17.07%) Amazon Redshift

Data Pipeline
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 328 (34.56%) Apache Spark
2 134 (14.12%) Apache Airflow
3 91 (9.59%) Apache
4 60 (6.32%) Elasticsearch
5 29 (3.06%) Confluence
6 21 (2.21%) Luigi
7 13 (1.37%) Apache Pulsar
8 11 (1.16%) Apache Pig
9 7 (0.74%) Oracle Workflow
10 6 (0.63%) Tornado
11 5 (0.53%) Ethereum
12 3 (0.32%) SAS
13 2 (0.21%) Oracle Application Server
14 1 (0.11%) IIS
14 1 (0.11%) OpenStack
Applications
1 19 (2.00%) Microsoft Excel
2 8 (0.84%) Spreadsheet
3 5 (0.53%) Adobe Illustrator
3 5 (0.53%) Fireworks
3 5 (0.53%) Photoshop
4 4 (0.42%) Microsoft Office
4 4 (0.42%) Microsoft PowerPoint
5 1 (0.11%) Chatbot
Business Applications
1 6 (0.63%) Eloqua
2 4 (0.42%) Summit
3 2 (0.21%) Dynamics CRM
3 2 (0.21%) Oracle Procure-to-Pay
3 2 (0.21%) SAP R/3
4 1 (0.11%) Markit EDM
Cloud Services
1 578 (60.91%) AWS
2 302 (31.82%) GCP
3 267 (28.13%) Azure
4 168 (17.70%) Snowflake
5 159 (16.75%) Serverless
6 118 (12.43%) Power Platform
7 114 (12.01%) Amazon S3
8 105 (11.06%) AWS Lambda
9 98 (10.33%) Amazon EMR
10 65 (6.85%) Amazon Kinesis
11 55 (5.80%) Amazon SQS
12 52 (5.48%) GitHub
13 44 (4.64%) Amazon EC2
13 44 (4.64%) IBM Cloud
14 36 (3.79%) Amazon ECS
14 36 (3.79%) Azure Data Factory
15 35 (3.69%) AWS Data Pipeline
16 31 (3.27%) Google Compute Engine
16 31 (3.27%) SaaS
17 30 (3.16%) AWS CloudFormation
Communications & Networking
1 21 (2.21%) Internet
2 15 (1.58%) DNS
3 14 (1.48%) TCP/IP
4 8 (0.84%) DHCP
4 8 (0.84%) HTTPS
4 8 (0.84%) IPv4
4 8 (0.84%) IPv6
4 8 (0.84%) SMTP
4 8 (0.84%) SNMP
4 8 (0.84%) SSH
5 6 (0.63%) SMS
6 4 (0.42%) Broadband
6 4 (0.42%) HTTP
7 3 (0.32%) 5G
8 2 (0.21%) Network Security
Database & Business Intelligence
1 300 (31.61%) Big Data
2 257 (27.08%) Data Warehouse
3 184 (19.39%) NoSQL
4 173 (18.23%) Hadoop
5 164 (17.28%) Data Lake
6 162 (17.07%) Amazon Redshift
7 145 (15.28%) PostgreSQL
8 123 (12.96%) Relational Database
9 118 (12.43%) Power BI
10 115 (12.12%) Tableau
11 98 (10.33%) BigQuery
12 75 (7.90%) MySQL
13 74 (7.80%) Electronic Patient Records
13 74 (7.80%) Looker
14 72 (7.59%) Amazon Athena
15 59 (6.22%) MongoDB
16 56 (5.90%) Amazon RDS
16 56 (5.90%) Neo4j
17 45 (4.74%) Amazon DynamoDB
18 39 (4.11%) Redis
Development Applications
1 119 (12.54%) Git (software)
2 81 (8.54%) Jenkins
3 46 (4.85%) JIRA
4 44 (4.64%) Atlassian Bamboo
4 44 (4.64%) Vagrant
5 29 (3.06%) CircleCI
6 18 (1.90%) GitLab
7 17 (1.79%) Terragrunt
8 16 (1.69%) git-flow
9 7 (0.74%) MLflow
10 6 (0.63%) Oracle APEX
10 6 (0.63%) Spinnaker
11 5 (0.53%) Gradle
12 3 (0.32%) Bitbucket
12 3 (0.32%) Cucumber
12 3 (0.32%) Selenium
12 3 (0.32%) SoapUI
13 2 (0.21%) Artifactory
13 2 (0.21%) Subversion
13 2 (0.21%) TeamCity
General
1 208 (21.92%) Social Skills
2 156 (16.44%) Finance
3 128 (13.49%) Analytical Skills
4 57 (6.01%) Marketing
5 44 (4.64%) Banking
6 42 (4.43%) Retail
7 38 (4.00%) Games
8 28 (2.95%) Law
9 26 (2.74%) Legal
10 25 (2.63%) Manufacturing
10 25 (2.63%) Publishing
11 23 (2.42%) Public Sector
12 20 (2.11%) Advertising
13 15 (1.58%) Telecoms
14 9 (0.95%) Czech Language
15 7 (0.74%) Electronics
16 6 (0.63%) Automotive
17 5 (0.53%) Billing
17 5 (0.53%) Pharmaceutical
17 5 (0.53%) Spanish Language
Job Titles
1 436 (45.94%) Data Engineer
2 119 (12.54%) Developer
3 109 (11.49%) Python Engineer
4 97 (10.22%) Senior Data Engineer
5 83 (8.75%) Architect
6 67 (7.06%) Data Architect
7 54 (5.69%) AWS Engineer
8 46 (4.85%) Software Engineer
9 44 (4.64%) Big Data Engineer
10 38 (4.00%) Platform Engineer
11 36 (3.79%) Data Scientist
12 34 (3.58%) Analyst
13 30 (3.16%) Azure Engineer
13 30 (3.16%) Data Analyst
14 28 (2.95%) Machine Learning Engineer
15 27 (2.85%) Cloud Engineer
15 27 (2.85%) Platform Architect
15 27 (2.85%) Python Developer
16 21 (2.21%) Full Stack Developer
17 20 (2.11%) DevOps Engineer
Libraries, Frameworks & Software Standards
1 216 (22.76%) Kafka
2 112 (11.80%) Node.js
3 91 (9.59%) React
4 57 (6.01%) PySpark
5 56 (5.90%) Django
6 55 (5.80%) Spark Streaming
7 54 (5.69%) GraphQL
8 53 (5.58%) HTML
9 49 (5.16%) scikit-learn
10 46 (4.85%) REST
11 41 (4.32%) Apache Beam
11 41 (4.32%) Pandas
11 41 (4.32%) TensorFlow
12 35 (3.69%) CSS
12 35 (3.69%) Flask
13 30 (3.16%) AngularJS
13 30 (3.16%) Web Services
14 27 (2.85%) PyTorch
15 23 (2.42%) NumPy
16 19 (2.00%) SciPy
Miscellaneous
1 61 (6.43%) Algorithms
2 52 (5.48%) Data Structures
3 36 (3.79%) Self-Motivation
4 32 (3.37%) Public Cloud
5 31 (3.27%) Cloud Native
6 21 (2.21%) Greenfield Project
6 21 (2.21%) User Experience
7 19 (2.00%) Social Media
8 18 (1.90%) Hybrid Cloud
8 18 (1.90%) Mobile Games
9 16 (1.69%) Renewable Energy
10 14 (1.48%) Management Information System
11 13 (1.37%) Data Centre
11 13 (1.37%) Mobile App
12 10 (1.05%) Virtual Team
13 9 (0.95%) Enterprise Software
14 8 (0.84%) Digital Media
14 8 (0.84%) Internet of Things
14 8 (0.84%) Video\Live Streaming
15 7 (0.74%) Foreign Exchange (FX)
Operating Systems
1 70 (7.38%) Linux
2 37 (3.90%) Unix
3 6 (0.63%) Apple iOS
4 2 (0.21%) Windows
5 1 (0.11%) CentOS
5 1 (0.11%) Red Hat Enterprise Linux
Processes & Methodologies
1 477 (50.26%) Analytics
2 407 (42.89%) Data Science
3 353 (37.20%) ETL
4 311 (32.77%) Data Engineering
5 300 (31.61%) Agile Software Development
6 243 (25.61%) Software Engineering
7 220 (23.18%) Data Modelling
8 204 (21.50%) Artificial Intelligence
9 203 (21.39%) Machine Learning
10 201 (21.18%) Business Intelligence
11 183 (19.28%) DevOps
12 154 (16.23%) Problem-Solving
13 153 (16.12%) Visualisation
14 124 (13.07%) Mentoring
15 123 (12.96%) Microservices
16 113 (11.91%) Continuous Integration
17 111 (11.70%) Computer Science
17 111 (11.70%) Data Management
18 102 (10.75%) Data Architecture
19 81 (8.54%) Data Visualisation
Programming Languages
1 750 (79.03%) Python
2 554 (58.38%) SQL
3 206 (21.71%) Java
4 107 (11.28%) Scala
5 103 (10.85%) JavaScript
6 63 (6.64%) R
7 55 (5.80%) Go
8 52 (5.48%) TypeScript
9 43 (4.53%) C#
10 32 (3.37%) Bash Shell
11 29 (3.06%) Rust
12 28 (2.95%) Shell Script
13 21 (2.21%) Ruby
14 19 (2.00%) C++
15 18 (1.90%) PowerShell
16 17 (1.79%) T-SQL
17 12 (1.26%) C
18 7 (0.74%) LookML
18 7 (0.74%) PL/SQL
19 6 (0.63%) Apple Swift
Qualifications
1 122 (12.86%) Degree
2 35 (3.69%) Computer Science Degree
3 21 (2.21%) Master's Degree
4 10 (1.05%) AWS Certification
5 6 (0.63%) Numerate Degree
6 4 (0.42%) SC Cleared
6 4 (0.42%) Security Cleared
7 3 (0.32%) PhD
8 1 (0.11%) BPSS Clearance
8 1 (0.11%) IBM Certification
8 1 (0.11%) Microsoft Certification
Quality Assurance & Compliance
1 105 (11.06%) Data Quality
2 68 (7.17%) QA
3 23 (2.42%) GDPR
4 11 (1.16%) Accessibility
4 11 (1.16%) SLA
5 2 (0.21%) GRC
5 2 (0.21%) MiFID
6 1 (0.11%) ISO/IEC 27001
System Software
1 250 (26.34%) Docker
2 44 (4.64%) VirtualBox
3 5 (0.53%) Apache ZooKeeper
4 2 (0.21%) Apache Flume
4 2 (0.21%) HDFS
Systems Management
1 201 (21.18%) Kubernetes
2 146 (15.38%) Terraform
3 77 (8.11%) Ansible
4 44 (4.64%) Docker Swarm
5 15 (1.58%) Puppet
6 11 (1.16%) Opscode Chef
7 8 (0.84%) Hadoop YARN
8 7 (0.74%) Oozie
9 6 (0.63%) Istio
10 5 (0.53%) Kubeflow
11 2 (0.21%) Ambari
11 2 (0.21%) Kibana
11 2 (0.21%) logstash
12 1 (0.11%) Control-M
12 1 (0.11%) Grafana
12 1 (0.11%) Mesos
12 1 (0.11%) Prometheus
Vendors
1 166 (17.49%) Google
2 71 (7.48%) Talend
3 69 (7.27%) Databricks
4 57 (6.01%) Oracle
5 55 (5.80%) Informatica
6 53 (5.58%) Hortonworks
7 52 (5.48%) Cloudera
8 46 (4.85%) IBM
9 36 (3.79%) Microsoft
10 14 (1.48%) Adobe
10 14 (1.48%) SAP
11 11 (1.16%) Dell
12 10 (1.05%) Teradata
13 8 (0.84%) Cognos
13 8 (0.84%) Monetate
13 8 (0.84%) New Relic
14 6 (0.63%) Alteryx
14 6 (0.63%) Instagram
14 6 (0.63%) Marketo
14 6 (0.63%) Splunk