Senior Automation Engineer (Python, LLM, Spark, ETL)

Senior Automation Engineer (Python, LLM, Spark, ETL)

Role Overview

• We are seeking a highly skilled Senior Automation Engineer with expertise in Python, Large Language Models (LLMs), Spark, and ETL development.
• The role focuses on building intelligent automation solutions, scalable data pipelines, and integrating AI capabilities into enterprise workflows.

Automation & Python Development

• Design and develop automation frameworks using Python.
• Build reusable scripts for data processing and operational tasks.
• Develop APIs and integration services.

LLM & AI Integration

• Implement LLM-based automation solutions.
• Integrate AI models for data enrichment and automation.
• Develop and optimize prompt engineering strategies.

Big Data Processing (Spark)

• Develop scalable pipelines using Apache Spark (PySpark/Scala).
• Handle large-scale data processing with performance optimization.
• Implement batch and near real-time processing.

ETL & Data Engineering

• Develop ETL/ELT pipelines for multiple data sources.
• Ensure data quality, integrity, and consistency.
• Work with structured and semi-structured data formats.

CI/CD & DevOps

• Implement CI/CD pipelines using modern tools.
• Manage version control and deployments.
• Set up monitoring and alerting mechanisms.

Required Qualifications

• 6–10+ years of experience in automation or data engineering.
• Strong Python programming skills.
• Hands-on experience with LLMs and Spark.
• Experience in ETL pipeline development.

Preferred Qualifications

• Experience with GCP Cloud platforms.
• Exposure to Teradata systems.
• Knowledge of orchestration tools like Airflow.
• Familiarity with Docker and Kubernetes.