Milestone 4.5.B.1 — Distribution-Aware Drift Tests

Status: Planned
Goal: Track B — Drift Detection
Phase: 4.5 — Intelligence Layer
Estimated effort: 4 days
Track: Track B — Drift Detection
ADR required: None (statistical methodology documented in ADR-0045)

Why This Milestone Exists

The original plan's drift detection uses a single fixed z-score/cosine/KL divergence triple against a static baseline. Each of these has a known failure mode:

Z-score assumes Gaussian distribution — token counts are right-skewed
Static baseline cannot distinguish gradual drift from a sudden break
Single embedding centroid destroys multi-modal behavior structure
KL divergence is undefined/infinite when a new tool type appears with zero baseline probability

This milestone replaces all three with feature-appropriate tests that operate on the richer statistical representations stored in 4.5.A.1/A.2.

Non-Goals

Threshold values (4.5.B.2 calibrates these via empirical procedure)
Severity aggregation and action policy (4.5.B.3)
Dashboard alert display (4.5.C.4)

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

Fingerprinting / drift

Suggested naming (provisional)

Rename freely to match the change that actually lands.

Branch: feature/m4-5-b1-distribution-aware-drift-tests
PR title: feat(worker/drift): distribution-aware drift tests with ADWIN and KS/JS/chi2 (m4.5.B.1)

Deliverables

Target outcomes for the milestone; concrete artifacts may differ from any sketch above.

Test selection per feature class

Feature class	Test	Why chosen over original
Quantile-sketch features (tokens, length)	Kolmogorov-Smirnov two-sample test between baseline and current t-digest-approximated distributions	KS compares whole distributions, not just first two moments — catches shape changes z-score misses entirely
Tool distribution	Jensen-Shannon divergence with calibrated threshold (B.2)	Symmetric, bounded [0, ln 2], no infinite-blowup edge case, combined with Laplace smoothing
Response semantics	Cluster-membership shift via chi-squared test comparing per-cluster assignment weights	Detects "a new behavior mode appeared" directly vs. inferring from a moved centroid
Error/timeout rate	Beta-Binomial credible-interval overlap check	Correctly accounts for sample size; 3/10 errors ≠ 300/1000 at the same alert severity

Sequential detector: ADWIN

Layer river's ADWIN (Adaptive Windowing) on top of the per-window statistical tests. ADWIN automatically maintains a variable-size window and only flags drift when there's statistical evidence the distribution changed — handles both gradual drift and sudden breaks by design, with a proven false-positive rate bound.

services/worker/
 drift/
 __init__.py
 detectors.py # KS test, JS divergence, chi-squared, ADWIN wrapper (river)
 calibration.py # offline calibration script (used in B.2)
 thresholds.yaml # calibrated thresholds (populated in B.2)
 tasks.py # Celery task: detect_drift(agent_id, fingerprint_id)
 tests/
 test_detectors.py
 test_calibration.py

Dependencies added

river (Python, BSD-3) — ADWIN, Page-Hinkley, DDM/EDDM concept-drift detectors, battle-tested in streaming ML
scipy — KS two-sample test, chi-squared test (likely already present in ML stack; confirm)

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

KS two-sample test correctly detects a distribution shift on t-digest-approximated quantile data
JS divergence with Laplace smoothing: new tool type appearing (zero baseline probability) produces a finite, meaningful score — not undefined or infinite
Chi-squared test correctly flags "new behavior cluster appeared" in a synthetic fixture that the original single-centroid cosine approach would have missed — this regression test must explicitly document the comparison
ADWIN integrated via river; unit-tested against synthetic drift/no-drift streams with known drift injection points
All detectors run fully async, off the proxy hot path — verified by test that no detector import appears in proxy-service code
river added to services/worker/requirements.txt

Prerequisites

4.5.A.1, 4.5.A.2 merged (feature vectors and schema exist)
Phase 3.5 gold-set conversations available for calibration in B.2