disaster-recovery drills for stream and batch environments. Architecture & Automation Collaborate with data engineering and product teams to architect scalable, fault-tolerant pipelines using AWS services (e.g., Step Functions , EMR , Lambda , Redshift ) integrated with Apache Flink and Kafka . Troubleshoot & Maintain Python -based applications. Harden CI/CD for data jobs: implement automated testing of data schemas, versioned Flink More ❯
surveillance platform, including TT Trade Surveillance , which leverages machine learning to detect trading behavior that may trigger regulatory inquiries. Our tech stack includes Java, Python, Apache Spark (on Serverless EMR), AWS Lambda, DynamoDB, S3, SNS/SQS , and other cloud-native services. As part of a high-impact engineering team, you'll help design and implement fault-tolerant data … reliability are critical. What Will You be Involved With? Design and build scalable, distributed systems using Java , Python , and Apache Spark Develop and optimize Spark jobs on AWS Serverless EMR for processing large-scale time-series datasets Build event-driven and batch processing workflows using Lambda , SNS/SQS , and DynamoDB Read, process, and transform data from a variety … Strong programming skills in Java (multithreading, concurrency, performance tuning) Deep experience with Apache Spark and Spark Streaming Proficiency with AWS services , including Lambda, DynamoDB, S3, SNS, SQS, and Serverless EMR Experience with SQL and NoSQL databases Hands-on with Python , especially in data handling (pandas, scikit-learn, etc.) Familiarity with RESTful web services and event-driven architectures Strong debugging More ❯
Develop a data validation framework to automate quality control processes Requirements Over 3+ years of hands-on Python development experience for production systems Proficient in data cloud computing technologies (EMR, Databricks) within leading cloud platforms (AWS, Google Cloud, or Azure) Skilled in designing and implementing data pipelines using distributed storage platforms Demonstrated analytical skills and structured approach to software More ❯
