Milestone 3.5.C.3 — Composite Scorer v2 (Relevance-Gated, Category-Conditional Decay)

Status: Planned
Goal: Track C — Context Assembly Engine
Phase: 3.5 — Extraction & Context Assembly
Estimated effort: 2 days
Track: Track C — Context Assembly Engine
ADR required: ADR-0040 (rewritten) — Memory scoring formula v2 Depends on: 3.5.C.2

Why This Milestone Exists

The original design's own scorer test documents its own flaw and just shrugs at it:

"test_relevance_dominates: ... First must score higher (0.40 > 0.25+0.20+0.10+0.05 = 0.60... wait actually 0.40 < 0.60 — this means a very old, frequently accessed, high-confidence memory can outscore a highly relevant one). Document this in ADR."

That's not a documented tradeoff — it's a bug being pre-emptively excused. A memory scorer where irrelevant-but-old-and-frequently-accessed content can outrank the single most relevant memory to the current query is a retrieval-quality regression by design. This redesign fixes it with a relevance gate, not just a reweighting.

Non-Goals

Packing (3.5.C.4)
Changing the proto wire contract

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

Context assembly service
Tokenizer registry / counting

Suggested naming (provisional)

Rename freely to match the change that actually lands.

Branch: feature/m3-5-c3-composite-scorer-v2
PR title: fix(context): composite scorer v2 with relevance gate and category-conditional decay (m3.5.C.3)

Design

Python

# Illustrative — exact path may differ
 
RELEVANCE_FLOOR = 0.15 # memories below this cosine similarity are excluded before scoring, period
 
# Category-conditional half-lives (days) — replaces the single global λ=0.05
CATEGORY_HALF_LIFE_DAYS: dict[str, float] = {
 "factual": 180.0, # "user's DB is Postgres" — barely decays
 "procedural": 365.0, # "how to deploy" — decays even slower; correctness-critical
 "preference": 90.0,
 "behavioral": 45.0,
 "episodic": 14.0, # original λ=0.05 value, now scoped to where it's actually correct
}
 
class MemoryScorer:
 def score(self, memories: list[dict], query_embedding: list[float]) -> list[ScoredMemory]:
 # Gate FIRST: never let a stale-but-frequent memory outcompete a relevant one
 # by simply not scoring irrelevant candidates at all.
 candidates = [m for m in memories if m.get("similarity", 0.0) >= RELEVANCE_FLOOR]
 scored = [self._score_one(m) for m in candidates]
 scored.sort(key=lambda m: m.composite_score, reverse=True)
 return scored
 
 def _recency(self, memory: dict) -> float:
 half_life = CATEGORY_HALF_LIFE_DAYS.get(memory.get("category", "episodic"), 14.0)
 lam = math.log(2) / half_life
 age_days = _age_days(memory.get("created_at"))
 return math.exp(-lam * age_days)

Why a Hard Relevance Floor Instead of Just Reweighting

Reweighting (e.g., bumping relevance to 0.60) still leaves an unbounded failure mode — with enough accumulated retrieval_count and confidence, a stale memory can eventually cross any fixed weight threshold. A gate is a hard invariant: irrelevant content structurally cannot be scored, let alone win. This is the standard pattern used by real retrieval-ranking systems (BM25/vector prefilter → rerank), not a novel idea — it's what the original design should have had from the start.

ADR-0040 (Rewritten) — Memory Scoring Formula v2

Must document, verbatim as an ADR section:

The relevance-floor rationale and the exact failure mode it prevents (cite the original test's own admission as the motivating bug report).
Category half-life table with justification per category (reference the retrieval-quality gold-set benchmark from Phase 3 Track B as the empirical validation source).
Formula weights unchanged (0.40/0.25/0.20/0.10/0.05) — validated, not redesigned, since the gate is what fixes the failure mode, not the weights themselves.
Explicit compatibility note: AssemblyOptions per-request weight overrides now apply after the relevance gate; callers cannot use weight overrides to bypass the gate.

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

RELEVANCE_FLOOR gate implemented and unit-tested: a memory with similarity=0.05, confidence=1.0, retrieval_count=1000 must score zero / be excluded, not merely rank low
Category half-life table implemented; unit test proves a 60-day-old factual memory scores higher on recency than a 60-day-old episodic memory
Regression test against the Phase 3 Track B retrieval-quality gold set — this scorer change should not decrease precision@k on the committed baseline
ADR-0040 rewritten and cross-linked from this milestone