Data Engineer
HCLTech is a global technology company, home to more than 220,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services.
Mandatory Required Skills
Python / PySpark / Scala, SQL and data warehousing concepts
Preferred /Desired Skills
- Design and build data pipelines (ETL/ELT)
- Manage data infrastructure (databases, lakes, warehouses)
- Handle large-scale data processing and transformation
- Ensure data quality, validation, and reliability
- Integrate data from multiple systems/APIs
Detailed Job Description
- Design and develop end-to-end data pipelines (ETL/ELT) for batch and real-time processing.
- Build and maintain data ingestion frameworks from multiple data sources.
- Develop scalable data models and data warehouse solutions.
- Implement workflow orchestration (Airflow / Cloud Composer) including retries, dependencies, and monitoring.
- Ensure data quality, validation, and governance mechanisms.
- Design and manage streaming pipelines (Kafka / Pub-Sub) for near real-time data processing.
- Optimize data pipelines for performance, scalability, and cost efficiency.
- Monitor, troubleshoot, and enhance pipeline reliability and observability.
- Collaborate with business, analytics, and ML teams to translate requirements into data solutions.