Period
to 19 October 2021

The following table provides summary statistics for permanent job vacancies advertised in the City of London with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 19 October 2021 with a comparison to the same period in the previous 2 years.

Data Pipelines
Central London > City of London
6 months to
19 Oct 2021
Same period 2020 Same period 2019
Rank 130 169 233
Rank change year-on-year +39 +64 +69
Permanent jobs citing Data Pipeline 133 31 88
As % of all permanent jobs advertised in the City of London 2.23% 0.86% 0.88%
As % of the Processes & Methodologies category 2.36% 0.91% 0.94%
Number of salaries quoted 126 26 84
10th Percentile £49,375 £45,625 £43,750
25th Percentile £57,500 £57,625 £48,750
Median annual salary (50th Percentile) £72,500 £71,250 £66,250
Median % change year-on-year +1.75% +7.55% -1.85%
75th Percentile £83,750 £85,000 £78,750
90th Percentile £92,500 £96,875 £95,000
Central London median annual salary £72,500 £72,500 £70,000
% change year-on-year - +3.57% +3.70%

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies advertised in the City of London with a requirement for process or methodology skills.

All Process and Methodology Skills
City of London
Permanent vacancies with a requirement for process or methodology skills 5,635 3,407 9,350
As % of all permanent IT jobs advertised in the City of London 94.59% 94.25% 93.91%
Number of salaries quoted 5,024 2,996 8,237
10th Percentile £43,250 £45,000 £38,000
25th Percentile £55,000 £53,750 £51,250
Median annual salary (50th Percentile) £72,500 £70,000 £67,500
Median % change year-on-year +3.57% +3.70% +3.85%
75th Percentile £91,250 £85,875 £82,500
90th Percentile £107,500 £100,625 £97,500
Central London median annual salary £70,000 £70,000 £65,000
% change year-on-year - +7.69% +4.00%

Data Pipeline
Job Vacancy Trend in the City of London

Job postings citing Data Pipeline as a proportion of all IT jobs advertised in the City of London.

Job vacancy trend for Data Pipeline in the City of London

Data Pipeline
Salary Trend in the City of London

3-month moving average salary quoted in jobs citing Data Pipeline in the City of London.

Salary trend for Data Pipeline in the City of London

Data Pipeline
Salary Histogram in the City of London

Salary distribution for jobs citing Data Pipeline in the City of London over the 6 months to 19 October 2021.

Salary histogram for Data Pipeline in the City of London

For the 6 months to 19 October 2021, IT jobs citing Data Pipeline also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads across the City of London region with a requirement for Data Pipeline.

1 90 (67.67%) SQL
1 90 (67.67%) Python
2 71 (53.38%) AWS
3 58 (43.61%) Analytics
4 48 (36.09%) Data Science
5 43 (32.33%) Apache Spark
6 42 (31.58%) Big Data
6 42 (31.58%) Data Warehouse
6 42 (31.58%) Java
7 35 (26.32%) Data Modelling
7 35 (26.32%) Data Lake
8 34 (25.56%) Azure
9 32 (24.06%) Data Engineering
10 31 (23.31%) Agile Software Development
11 28 (21.05%) Amazon Redshift
12 27 (20.30%) ETL
13 25 (18.80%) Finance
13 25 (18.80%) Hadoop
13 25 (18.80%) NoSQL
13 25 (18.80%) Problem-Solving
14 24 (18.05%) Data Quality
14 24 (18.05%) Kafka
14 24 (18.05%) Relational Database
14 24 (18.05%) Degree
14 24 (18.05%) Computer Science
15 23 (17.29%) DevOps
15 23 (17.29%) Scala
16 21 (15.79%) Amazon EC2
17 20 (15.04%) Business Intelligence
18 19 (14.29%) Apache

Data Pipeline
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 43 (32.33%) Apache Spark
2 19 (14.29%) Apache
3 15 (11.28%) Apache Airflow
4 8 (6.02%) Elasticsearch
5 2 (1.50%) Apache Pig
5 2 (1.50%) Confluence
6 1 (0.75%) Luigi
Applications
1 10 (7.52%) Microsoft Excel
2 2 (1.50%) Microsoft Office
2 2 (1.50%) Microsoft PowerPoint
Cloud Services
1 71 (53.38%) AWS
2 34 (25.56%) Azure
3 21 (15.79%) Amazon EC2
4 17 (12.78%) GCP
5 16 (12.03%) Amazon S3
6 14 (10.53%) Power Platform
7 13 (9.77%) Serverless
7 13 (9.77%) Snowflake
8 11 (8.27%) AWS Lambda
9 10 (7.52%) AWS CloudFormation
10 9 (6.77%) Amazon EMR
11 7 (5.26%) Azure Data Factory
12 6 (4.51%) Amazon ECS
12 6 (4.51%) Amazon ELB
12 6 (4.51%) Amazon SQS
12 6 (4.51%) AWS Fargate
12 6 (4.51%) Virtual Private Cloud
13 5 (3.76%) Amazon EKS
13 5 (3.76%) AWS Data Pipeline
13 5 (3.76%) GitHub
Communications & Networking
1 6 (4.51%) DNS
1 6 (4.51%) TCP/IP
2 2 (1.50%) 5G
2 2 (1.50%) Broadband
3 1 (0.75%) Network Security
Database & Business Intelligence
1 42 (31.58%) Big Data
1 42 (31.58%) Data Warehouse
2 35 (26.32%) Data Lake
3 28 (21.05%) Amazon Redshift
4 25 (18.80%) Hadoop
4 25 (18.80%) NoSQL
5 24 (18.05%) Relational Database
6 15 (11.28%) Amazon RDS
6 15 (11.28%) MongoDB
7 14 (10.53%) Power BI
7 14 (10.53%) Tableau
8 12 (9.02%) Looker
9 7 (5.26%) Spark SQL
10 6 (4.51%) BigQuery
10 6 (4.51%) Metadata
10 6 (4.51%) MySQL
11 5 (3.76%) Amazon Aurora
11 5 (3.76%) Blockchain
11 5 (3.76%) Electronic Patient Records
11 5 (3.76%) PostgreSQL
Development Applications
1 11 (8.27%) Git (software)
1 11 (8.27%) JIRA
2 6 (4.51%) Gradle
2 6 (4.51%) Terragrunt
3 5 (3.76%) git-flow
3 5 (3.76%) Jenkins
4 2 (1.50%) ER Studio
4 2 (1.50%) ERwin
4 2 (1.50%) GitLab
4 2 (1.50%) PowerDesigner
4 2 (1.50%) Subversion
5 1 (0.75%) Artifactory
5 1 (0.75%) Atlassian Bamboo
5 1 (0.75%) Maven
5 1 (0.75%) Octopus Deploy
5 1 (0.75%) Selenium
5 1 (0.75%) Sonatype Nexus
5 1 (0.75%) TeamCity
5 1 (0.75%) Vagrant
General
1 25 (18.80%) Finance
2 16 (12.03%) Analytical Skills
2 16 (12.03%) Games
2 16 (12.03%) Social Skills
3 11 (8.27%) Banking
4 9 (6.77%) Marketing
5 7 (5.26%) Legal
5 7 (5.26%) Publishing
6 5 (3.76%) Retail
7 2 (1.50%) Advertising
7 2 (1.50%) Telecoms
8 1 (0.75%) Financial Institution
8 1 (0.75%) Law
8 1 (0.75%) Manufacturing
Job Titles
1 89 (66.92%) Data Engineer
2 17 (12.78%) Senior Data Engineer
3 16 (12.03%) AWS Engineer
4 14 (10.53%) Big Data Engineer
5 12 (9.02%) Data Scientist
6 10 (7.52%) Developer
7 8 (6.02%) Hadoop Engineer
7 8 (6.02%) Java Engineer
8 7 (5.26%) .NET Developer
9 6 (4.51%) DevOps Engineer
9 6 (4.51%) Junior
9 6 (4.51%) Junior Data Scientist
9 6 (4.51%) Scala Engineer
10 5 (3.76%) Senior Data Scientist
10 5 (3.76%) Senior Scala Engineer
11 4 (3.01%) Engineering Manager
11 4 (3.01%) Operations Engineer
12 3 (2.26%) Analyst
12 3 (2.26%) Architect
12 3 (2.26%) Senior Developer
Libraries, Frameworks & Software Standards
1 24 (18.05%) Kafka
2 9 (6.77%) PySpark
3 7 (5.26%) .NET
4 6 (4.51%) AngularJS
4 6 (4.51%) JSON
5 5 (3.76%) .NET Core
6 2 (1.50%) Apache Avro
6 2 (1.50%) Middleware
6 2 (1.50%) Protocol Buffers
6 2 (1.50%) RabbitMQ
6 2 (1.50%) XML
6 2 (1.50%) YAML
7 1 (0.75%) CSS
7 1 (0.75%) Flask
7 1 (0.75%) HTML
7 1 (0.75%) Pandas
7 1 (0.75%) React Native
7 1 (0.75%) RESTful
7 1 (0.75%) Spark Streaming
7 1 (0.75%) Web Services
Miscellaneous
1 11 (8.27%) Mobile Games
2 9 (6.77%) Algorithms
3 8 (6.02%) Data Centre
3 8 (6.02%) Data Structures
4 7 (5.26%) Social Media
5 6 (4.51%) Greenfield Project
5 6 (4.51%) Legacy Code
6 5 (3.76%) Client/Server
7 3 (2.26%) Enterprise Software
7 3 (2.26%) Public Cloud
8 2 (1.50%) Derivative
9 1 (0.75%) Acoustics
9 1 (0.75%) Analytical Mindset
9 1 (0.75%) Blog
9 1 (0.75%) e-Learning
9 1 (0.75%) Foreign Exchange (FX)
9 1 (0.75%) User Experience
Operating Systems
1 8 (6.02%) Linux
Processes & Methodologies
1 58 (43.61%) Analytics
2 48 (36.09%) Data Science
3 35 (26.32%) Data Modelling
4 32 (24.06%) Data Engineering
5 31 (23.31%) Agile Software Development
6 27 (20.30%) ETL
7 25 (18.80%) Problem-Solving
8 24 (18.05%) Computer Science
9 23 (17.29%) DevOps
10 20 (15.04%) Business Intelligence
11 19 (14.29%) DataOps
11 19 (14.29%) Machine Learning
12 17 (12.78%) Data Transformation
12 17 (12.78%) Scrum
13 16 (12.03%) Software Engineering
14 15 (11.28%) Data Analysis
15 14 (10.53%) Data Architecture
15 14 (10.53%) Fintech
15 14 (10.53%) Predictive Modelling
16 13 (9.77%) Model Validation
Programming Languages
1 90 (67.67%) Python
1 90 (67.67%) SQL
2 42 (31.58%) Java
3 23 (17.29%) Scala
4 19 (14.29%) R
5 14 (10.53%) C#
6 4 (3.01%) Bash Shell
7 3 (2.26%) JavaScript
8 2 (1.50%) Go
8 2 (1.50%) PowerShell
9 1 (0.75%) C
9 1 (0.75%) C++
9 1 (0.75%) Haskell
Qualifications
1 24 (18.05%) Degree
2 6 (4.51%) AWS Certification
2 6 (4.51%) Master's Degree
3 3 (2.26%) Computer Science Degree
4 1 (0.75%) IBM Certification
4 1 (0.75%) PhD
Quality Assurance & Compliance
1 24 (18.05%) Data Quality
2 13 (9.77%) QA
System Software
1 4 (3.01%) Docker
2 2 (1.50%) Apache Flume
3 1 (0.75%) VirtualBox
Systems Management
1 13 (9.77%) Kubernetes
2 12 (9.02%) Terraform
3 5 (3.76%) Kubeflow
4 2 (1.50%) Ansible
5 1 (0.75%) Docker Swarm
5 1 (0.75%) Puppet
Vendors
1 12 (9.02%) Talend
2 9 (6.77%) Databricks
3 8 (6.02%) Oracle
4 6 (4.51%) Facebook
4 6 (4.51%) Google
5 4 (3.01%) Microsoft
6 3 (2.26%) Informatica
7 2 (1.50%) Alteryx
7 2 (1.50%) IBM
8 1 (0.75%) Cloudera
8 1 (0.75%) Hortonworks
8 1 (0.75%) Palantir
8 1 (0.75%) StreamSets
8 1 (0.75%) VMware