UPI Payments Compliance Lakehouse
5.07 million payment events, security at every layer, and a validator you can reconcile against ground truth.
The problem
Payments at PhonePe scale: every transaction must land, get cleaned, get PII-masked and be risk-queryable in minutes — with an audit trail a regulator would accept.
The build
An S3 medallion architecture with a Lambda schema validator gating ingestion, PII tokenized into a mapping table under a separate KMS key, gold risk and velocity tables serving RDS Postgres and DynamoDB. Seven Terraform modules provision dev and prod; GitHub Actions gates every change.
Architecture
— one teal highlight: the quarantine pathWhat I found
The validator intercepts exactly 2.00% of events — 101,449 of 5,074,342 — and cross-checks against the generator's manifest: 101,449 injected, 101,449 rejected, difference +0. That exactness cost two rounds of bugs the tests caught, including a manifest under-counting duplicates-of-malformed events by 1,522. A validator you cannot reconcile against ground truth is one you are trusting, not verifying.
The reps
The manifest bug was the interesting failure: the validator looked correct until the cross-check disagreed by 1,522 records, which meant the ground truth itself was wrong.
Results
- 5,074,342UPI events generated
- 2.00%intercepted before bronze, exactly reconciled
- 63tests across Lambda, Spark and DynamoDB
- 7Terraform modules, dev + prod
What's verified
Every case study carries this blockWhat was measured, and what was not. Nothing in the second column is estimated into the first.
Simulator, validator rate and exact cross-check, 39 Lambda tests via moto, 15 PySpark transform tests on real Spark, 9 DynamoDB tests, full-dataset idempotency proof, terraform validate.
No AWS account attached — no terraform apply, no real CloudTrail events, no Lambda running in Lambda, no Databricks job runs.