PySpark Developer - Financial Services
Essential skills/knowledge/experience:
- Experience of programming under Microsoft Azure Cloud Platform
- Have knowledge of Spark Programming Ability to write Spark code for large scale data processing, including RDDs, DataFrames, and Spark SQL
- Hands-on experience with lakehouses, dataflows, pipelines, and semantic models
- Ability to build ETL workflows
- Familiarity with time-series data, market feeds, transactional records, and risk metrics
- Familiarity with Git, DevOps pipelines, and automated deployment
- Strong communication skills with a collaborative mindset to work with and manage stakeholders
- Experience of programming under Microsoft Fabric Platform
- Python/Notebook programming
- PySpark programming
- Spark Streaming/batch processing
- Delta table Optimization
- Fabric Spark jobs
- Java programming language, OOP knowledge
- Database knowledge, including Relational Database and Non-SQL database.
- Experience of using the tools: Gitlab, Python unit test, CICD pipeline.
- Good skill of troubleshooting
- Familar with the Agile. Well communication.
- Good English listening and speaking for communicating requirements and development tasks/issues
Desirable skills/knowledge/experience:
- Ability to prepare and process datasets for Power BI usage
- Experience with OneLake, Azure Data Lake, and distributed computing environments
- Understanding of financial regulations such as GDPR, SOX etc.
- Spark application performance tuning
- Knowledge of Docker/Kubernetes