1% traffic sampling with live and candidate retrieval configs in parallel — divergence logged to ClickHouse, weekly automated report, dashboard panel for trend monitoring.
Milestone 5.C.3 — Production Shadow Evaluation
Status: Planned
Goal: Track C — Evaluation Harness Upgrade
Phase: 5 — Advanced Retrieval
Estimated effort: 3 days
Track: Track C — Evaluation Harness Upgrade
Why This Milestone Exists
Offline gold sets go stale and don't capture real traffic distribution. Shadow evaluation samples 1% of production retrievals, runs both live and candidate configs in parallel (fire-and-forget, zero latency impact), and logs divergence to catch regressions fixed gold sets structurally cannot.
Orientation (indicative)
Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.
During implementation, expect to:
- open the live tree and follow existing patterns before inventing new ones
- research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
- advance the design beyond the sketch where measurement or code reality says so
- land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it
Prefer outcomes over matching any particular file tree or command sequence.
Areas that may be involved (situational — not a checklist):
- Evaluation / benchmarks
- Hybrid retrieval
Success signals
Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.
- Shadow path adds zero latency to real requests (fire-and-forget, same pattern as async trace emitter)
- Dashboard panel showing divergence rate trend over 7/30 days
- Weekly automated report flags divergence exceeding threshold
Prerequisites
- Milestone 5.C.2 (Multi-mode benchmark runner)
- ClickHouse operational
Last updated on