PySpark Developer

PySpark Developer

Rate: £625 a day
Location: Remote
Clearance: CTC or above required

You will join a global IT consultancy delivering digital transformation to a public sector body.

Key Duties and Responsibilities

• Develop and optimise PySpark batch pipelines that process Parquet data and use Delta Lake for all IO, applying validation, enrichment, and billing calculation logic directly in PySpark code.
• Build reliable PySpark jobs that read/write Delta tables on ADLS Gen2.
• Implement in-code validations (schema, null/range/value checks, referential lookups), routing rejects to dedicated Delta "quarantine" tables.
• Design and implement billing logic - tariff/charge models, tiered pricing, pro-rata handling, VAT/discounts, adjustments, and full auditability.
• Externalise billing and validation rules via versioned JSON configs, ensuring deterministic, idempotent re-runs.
• Optimise Delta operations (MERGE, OPTIMIZE, Z-ORDER, VACUUM) and incremental/CDC merges into Azure SQL.
• Tune performance (partitioning, caching, broadcast joins) and maintain robust retries, checkpoints, and structured logging.
• Integrate with orchestrators (ADF or Container App Orchestrator) and CI/CD pipelines (GitHub Actions).
• Operate securely within private-network Azure environments (Managed Identity, RBAC, Private Endpoints).

Required Industry Knowledge and Competencies

• PySpark with Delta Lake (structured APIs, MERGE, schema evolution).
• Solid knowledge of Azure Synapse Spark pools or Databricks, ADLS Gen2, and Azure SQL.
• Strong engineering discipline: observability, retries, cost and performance optimisation.
• Great Expectations (for supplementary DQ checks).
• Familiarity with ADF orchestration and containerised Spark workloads.