Data Engineer - Energy Trading - London
Our client is a major commodity trader with 40 x offices across the globe.
We are recruiting a mid- senior Data Engineer with a background in Python, SQL, AWS to build data pipelines and dashboards that inform trading decisions daily. The Data Engineering team is responsible for a fundamental data system processing
50+ billion rows of data per day and feeding directly into trading decisions. The Data
Engineer will be designing, implementing and maintaining this system, keeping it
reliable, resilient, and low-latency.
We are looking for a highly technical engineer who is highly technical, with strong
experience working in MPP platforms and/or Spark, “big data” (e.g., weather forecasts,
AIS pings, satellite imagery, ...), and developing resilient and reliable data pipelines.
You will partner closely with business stakeholders and engineering teams to
understand their data requirements and deliver the necessary data infrastructure to
support their activities.
Essential Skills:
- 5+ years in the data engineering space
- Proficient with MPP Databases (Snowflake (preferred), Redshift, Big Query, Azure DW)
and/or Apache Spark
- Proficient at building resilient data pipelines for large datasets
- Deep AWS or cloud understanding across core and extended services.
- 2+ years experience working with at least 3 of the following: ECS, EKS, Lambda,
DynamoDB, Kinesis, AWS Batch, ElasticSearch/OpenSearch, EMR, Athena,
Docker/Kubernetes
- Proficient with Python and SQL, and with good experience with data modelling
- Experience with a modern orchestration tools (Airflow / Prefect / Dagster / similar)
- Comfortable working in a dynamic environment with evolving requirements
- A background in a dynamic financial services environment (Investment Bank, Hedge Fund)
- BSc in Computer Science from a Russell Group University
Desirable Skills:
- Exposure to trading and/or commodity business
- DBT experience
- Infrastructure as Code (Terraform, Cloud Formation, Ansible)
- CI/CD Pipelines (Jenkins / GIT / BitBucket)
- Database/SQL tuning skills
- Basic data science concepts