ADR-0043: Tokenizer registry architecture

• Status: Accepted
• Date: 2026-08-22
• Authors: IBEX Harness team
• Milestone: 2.5.G2.M1 Tokenizer registry

Context

ADR-0041 defines TokenizerFamily join keys (o200k_base, cl100k_base, claude, unknown on overlays). Phase 3.5 token budget calculation needs family-accurate pre-flight counts — using OpenAI tiktoken for Anthropic or self-hosted models is systematically wrong.

SPIKE findings (G2.M1):

Proxy hot path (ADR-0034) must not block on counting in G2 — registry construction + advisory /ready self-test only.

Decision

1) packages/tokenizer contract

type Tokenizer interface {
    Family() string
    Count(ctx context.Context, text string) (int, error)
}

• Registry maps TokenizerFamily → Tokenizer (mirrors provider.Registry pattern).
• NewLocalRegistry(assetDir) builds default v1 backends.
• ValidateCatalogCoverage(catalog, reg) fails closed when any catalog row (non-unknown family) lacks an impl.
• CountForModel(ctx, catalog, reg, model, text) resolves model → capability → family → Count.

2) Fail-closed startup, conservative runtime (Phase 3.5)

• Registry construction validates catalog coverage at proxy bootstrap.
• Missing family → startup error (ErrMissingTokenizer).
• Overlay with unknown → allowed in catalog validation; CountForModel returns error (no silent tiktoken substitution).
• Phase 3.5 may degrade to char estimate on remote failure — not in G2.

3) Air-gap asset policy

• o200k_base.tiktoken is committed under packages/tokenizer/assets/ (embedded via go:embed).
• cl100k_base loads from tiktoken-go-loader embedded assets.
• Optional IBEX_TOKENIZER_ASSET_DIR overrides BPE files by basename (operator-managed updates).
• No runtime Hugging Face downloads in v1.

4) Claude estimate (explicit, not exact)

claude backend implements Estimator with IsEstimate() == true. Formula:

tokens = 0                         if text is empty
tokens = ceil(rune_count / 3.5)    otherwise  (implemented as (runes*2+6)/7)

Consumers in Phase 3.5 may use Claude counts only after establishing a measured, versioned error bound against Anthropic count_tokens or an equivalent ground-truth source. Until then, apply family-specific safety buffers and do not treat the heuristic as directionally safe for budget enforcement.

5) Config surface (v1)

Deferred: IBEX_TOKENIZER_SERVICE_URL, IBEX_TOKENIZER_TIMEOUT_MS.

6) Proxy wiring (advisory)

• Bootstrap builds tokenizer registry after capability catalog merge.
• /ready advisory checker tokenizer: runs RunSelfTest on default vectors.
• Metrics: ibex_tokenizer_count_total{family,result}, ibex_tokenizer_count_duration_seconds{family} (observed when counting is invoked — not on Complete hot path in G2).

7) Input bounds and security

• Max input: 100 KiB (MaxCountTextBytes, aligned with ADR-0013 message limit).
• No raw text in logs — log family, model_id, text_len, duration, error class only.
• Counting is stateless; no tenant data in algorithm.

Consequences

Positive:

• Stable counting contract keyed by ADR-0041 families before Phase 3.5 budget math.
• OpenAI families match offline ground-truth vectors in CI.
• Air-gap deployments do not require OpenAI blob downloads at runtime.

Negative / follow-ups:

• Claude counts are estimates until API or service path lands.
• HF families (llama3, qwen2) require a follow-up milestone + allowlist extension in capability.go.
• Bundled o200k_base.tiktoken (~3.6 MiB) increases repo size; version bumps need changelog + vector re-verify.

References

• ADR-0041 Model capability registry
• ADR-0034 Performance methodology
• ENVIRONMENT_VARIABLES.md — IBEX_TOKENIZER_*