Milestone 5.C.2 — Multi-Mode Benchmark Runner

Status: Planned
Goal: Track C — Evaluation Harness Upgrade
Phase: 5 — Advanced Retrieval
Estimated effort: 3 days
Track: Track C — Evaluation Harness Upgrade

Why This Milestone Exists

Four retrieval configurations should be compared side by side to prove each stage adds value without degrading others. CI-gated regression prevents silent quality drops.

Configurations

Config	BM25	Rerank	Graph
`dense_only`	No	No	No
`hybrid`	Yes	No	No
`hybrid_rerank`	Yes	Yes	No
`full_graph`	Yes	Yes	Yes

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

Evaluation / benchmarks
Hybrid retrieval

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

CI job runs on every PR touching retrieval/reranker/graph code
full_graph should not regress precision@5 vs. hybrid_rerank by >2 points on non-graph queries
Reranker latency budget enforced as part of eval
Baseline results committed; any config regressing >3 points fails CI

Prerequisites

Milestone 5.C.1 (Gold set v2)
All Track A and B milestones