Parcel Intelligence Lakehouse on Azure
One pipeline, three fit-for-purpose stores, one number everyone agrees on.
The problem
India ships ~10 million e-commerce parcels a day, and three teams need the same scan trail in three incompatible ways: a customer wants a sub-50ms 'where is my order', ops wants hourly hub dashboards, finance wants exact reconciled history for SLA penalties. No single database serves all three.
The build
A medallion Delta lakehouse on Databricks, orchestrated by Azure Data Factory, transformed once and served three ways: Cosmos DB for point lookups, Azure SQL for the SLA mart, gold Delta for ops analytics. Silver handles dedup, repair-order reconstruction and PII masking; a control-total DQ gate sits between gold and the finance mart.
Architecture
— one teal highlight: the DQ gateWhat I found
Measured on 200,000 parcels: a pipeline that trusts arrival order reports 178,249 parcels delivered. The event-time-correct pipeline reports 190,014 — a 6.2% undercount, caused entirely by delivery scans that arrived after a later-syncing earlier scan. SLA penalties were being computed on the wrong number. The control-total check makes that disagreement impossible to publish.
The reps
Getting event-time ordering right meant rebuilding the silver dedup rule around repair order rather than ingest timestamp. The DQ gate was added after the divergence was found, not before — the gate exists because the bug did.
Results
- 200,000parcels simulated
- 1,604,238scan events processed
- 31,615duplicate scans removed
- 6.2%undercount caught by event-time correctness
What's verified
Every case study carries this blockWhat was measured, and what was not. Nothing in the second column is estimated into the first.
The simulator, the dedup counts, the divergence measurement and the control-total check — all reproducible with two commands.
No live Azure subscription attached; ADF pipelines and serving latency are not benchmarked against a running cluster.