Big Data Engineer
About the Role:
Synechron UK is seeking an experienced Lead Data Engineer to spearhead the development and evolution of our data engineering platforms. This leadership role requires a hands-on professional proficient in designing, building, and optimising enterprise-grade data solutions, with a focus on innovation, resilience, and regulatory compliance.
Key Responsibilities:
- Lead and develop robust data engineering platforms leveraging technologies such as Hadoop, Spark, and Splunk.
- Design, implement, and maintain scalable ETL/ELT data pipelines for diverse data types, including raw, structured, semi-structured, and unstructured data (SQL and NoSQL).
- Integrate large and disparate datasets using modern tools and frameworks to support analytical and operational needs.
- Collaborate effectively with BI and Analytics teams in dynamic environments, providing technical guidance and support.
- Develop, review, and maintain automated test plans, including unit and integration tests, to ensure high-quality, reliable code.
- Drive SRE principles within data engineering practices, ensuring service resilience, sustainability, and adherence to recovery time objectives.
- Support source control practices and implement CI/CD pipelines for continuous delivery of data solutions.
- Stay informed on current industry trends, regulatory requirements (cybersecurity, data privacy, data residency), and incorporate best practices into engineering processes.
- Represent Synechron in industry groups and vendor interactions to influence and adopt emerging technologies and standards.
Qualifications and Skills:
- Extensive experience with big data frameworks such as Hadoop, Spark, and Splunk.
- Strong scripting capabilities in Python, with experience in object-oriented and functional programming paradigms.
- Proven expertise in handling varied data formats and integrating large datasets across multiple platforms.
- Deep understanding of building, optimising, and managing complex ETL/ELT pipelines.
- Familiarity with version control systems and CI/CD tools.
- Experience working closely with BI and analytics teams, supporting data-driven decision-making.
- Excellent problem-solving skills with a data-driven mindset.
- Knowledge of agile methodologies (Scrum, Kanban).
- Ability to contribute in collaborative, fast-paced environments, including pair programming and team standups.
- Strong commitment to quality, test-driven development, and automation.