2 of 2 Contract AWS Step Functions Jobs in the City of London

implement ETL/ELT pipelines with strong error handling and retry logic. Develop incremental data processing patterns for large-scale datasets. Work with AWS services including Glue, Step Functions, S3, DynamoDB, Redshift, Lambda, and EventBridge. Build and optimise vector database solutions and embedding generation pipelines for semantic … Required Skills ETL/ELT pipeline design and data validation frameworks. Advanced Python (pandas, numpy, boto3) and SQL (complex queries, optimisation). Experience with AWS Glue , Step Functions , and event-driven architectures . Knowledge of vector databases , embeddings, and semantic search strategies. Familiarity with document parsing libraries ...

Required Technical Skills Data Pipeline & ETL Design, build, and maintain robust ETL/ELT pipelines for structured and unstructured data Hands-on experience with AWS Glue and AWS Step Functions Implementation of data validation, data quality frameworks, and reconciliation checks Strong error handling, monitoring, and retry … strategies in production pipelines Experience with incremental data processing patterns (CDC, watermarking, upserts) AWS Data Services Amazon S3: data lake architectures, partitioning strategies, lifecycle policies DynamoDB: data modeling, secondary indexes, streams, and performance optimization Amazon Redshift: foundational querying, integrations, and performance considerations AWS Lambda for scalable data processing ...