Qualifications • Designing and implementing real-time pipelines. • Designing and implementing data pipelines for CV/ML systems. • Experience with workflow management engines (i.e. Airflow, Luigi, Prefect, Dagster, digdag.io, Google Cloud Composer, AWS Step Functions, Azure Data Factory, UC4, Control-M). • Experience with data quality and validation. • Experience querying massive more »
FiveTran or Stitch Experience using Business Intelligence tools such as PowerBI, Looker or Tableau Experience with data pipeline tools such as DBT, Airflow or Luigi are a plus! Experience using cloud environments e.g. Azure or AWS Understanding of the Agile delivery method Working Conditions: · Permanent, London Chiswick Park · Working hours more »
London, England, United Kingdom Hybrid / WFH Options
Solirius Consulting
models, schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools and technologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc Implement data storage solutions using different types of databases, such as relational, non-relational, or cloud-based Working collaboratively with the client … SQL/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc Data acquisition and development of data sets and improving data more »
