Real-Time Quick-Commerce Operations Lakehouse
Streaming medallion with exactly-once ingestion — and the same job written twice to prove portability.
The problem
A dark store promising 10-minute delivery stocks out in minutes. Orders stream in every second and the ops dashboard has to show the problem now, not after the next batch window.
The build
Kafka (Redpanda) into Spark Structured Streaming with exactly-once bronze ingestion, watermarked late-event handling in silver, a dead-letter table with a reason per record, and 5-minute windowed gold with MERGE. The nightly backfill is written in both PySpark and Scala, and the same jar runs on GCP Dataproc into BigQuery — the portability claim, demonstrated rather than asserted.
Architecture
— one teal highlight: the dead-letter pathWhat I found
The streaming stockout detector fires when projected inventory runs out within 15 minutes, where an hourly batch would notice roughly 37 minutes later. That gap is arithmetic, not a benchmark — and the README says so. What is measured is the detector's edge cases: zero demand never divides by zero, at-zero alerts immediately, the 15.0-minute boundary alerts and 15.5 does not, all green in CI against real Spark.
The reps
Writing the same batch logic twice, in PySpark and Scala, surfaced assumptions the Python version had been quietly making about type coercion.
Results
- 3Kafka topics, 12 partitions
- 50/ssimulated event rate with rain surge
- 14producer coherence tests in CI
- 2clouds, one codebase
What's verified
Every case study carries this blockWhat was measured, and what was not. Nothing in the second column is estimated into the first.
Producer event coherence, detector edge cases, transform tests — all run against real Spark in CI on every PR.
No live broker, cluster, workspace or GCP project attached. The batch-vs-stream latency gap is stated as arithmetic, not a measured benchmark.