Milestone 3.5.B.4 — Extraction Quality Evaluation Harness

Status: Planned
Goal: Track B — Extraction Pipeline
Phase: 3.5 — Extraction & Context Assembly
Estimated effort: 3 days
Track: Track B — Extraction Pipeline
Depends on: 3.5.B.2

Why This Milestone Exists

The original plan's only quality signal was "extraction prompt produces valid JSON in > 99% of test cases" — that measures parseability, not correctness. There was no way to know if extraction actually captures the right facts, misses important ones, or hallucinates.

This milestone closes that gap, and it's the extraction-side counterpart to the retrieval-quality benchmark already planned for Phase 3's memory service.

Non-Goals

• Real-time extraction quality monitoring in the operator dashboard (Phase 4)
• A/B testing infrastructure for prompt variants

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

• open the live tree and follow existing patterns before inventing new ones
• research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
• advance the design beyond the sketch where measurement or code reality says so
• land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

• Extraction pipeline
• Workers / task runtime

Suggested naming (provisional)

Rename freely to match the change that actually lands.

• Branch: feature/m3-5-b4-extraction-eval-harness
• PR title: feat(worker): extraction quality evaluation harness with gold-set (m3.5.B.4)

Design

A versioned gold-set of ~100–200 hand-labeled synthetic conversations (multi-turn, covering all 5 categories, including intentionally ambiguous/multi-label cases and temporal-supersession cases), stored as fixture data:

services/worker/eval/
 gold_set/
 v1/
 conversations.jsonl # {conversation_id, turns: [...]}
 expected_memories.jsonl # {conversation_id, turn_index, expected: [ExtractedMemory]}
 baseline_results.json # committed baseline metrics per provider
 run_eval.py # computes precision/recall against gold set
 README.md # how to add new gold-set cases, versioning policy

Metrics

Extraction precision/recall per category: did the model extract the right memories for each category?

Category-assignment accuracy: did it multi-label correctly (not over-label, not under-label)?

Temporal-field accuracy: did it correctly identify supersession cases and set valid_until appropriately vs. leaving it as None for indefinite facts?

This becomes a CI-gated regression check — any prompt change or model swap should not drop below the committed baseline, mirroring the same "benchmark becomes a permanent regression gate" pattern established for retrieval quality in Phase 3.

CI Integration

run_eval.py is added to CI with a trigger on any PR that touches:

• extraction_provider.py
• extraction_schema.py
• The system prompt constants

Baseline tolerances (e.g., no regression of more than 3 percentage points on any metric) are committed to baseline_results.json.

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

• Gold set v1 committed with ≥100 conversations, reviewed by at least one human for label correctness
• run_eval.py produces precision/recall/category-accuracy report, runs in CI on every PR touching extraction_provider.py, extraction_schema.py, or the system prompt
• Baseline metrics committed to services/worker/eval/gold_set/v1/baseline_results.json; CI fails if a new run regresses any metric by more than the documented tolerance (3 percentage points)
• Both OpenAIExtractionProvider (gpt-4o-mini) and VLLMExtractionProvider (Qwen2.5-14B) evaluated against the same gold set, with results published side by side

Prerequisites

• 3.5.B.2 Cost-tiered extraction execution