Milestone 2.5.G2.M1 — Tokenizer Registry

Status: Completed (2.5.G2.M1 — ADR-0043) Goal: Track B — Tokenizer Registry
Phase: 2.5 — Provider Generalization & Foundation
Estimated effort: 3–4 days

Why This Milestone Exists

Token-budget math done with tiktoken (OpenAI's tokenizer) will be wrong for open-weight models — different vocabulary, merge rules, and special tokens. If this stays unfixed before Phase 3 context assembly is built, the budget calculator can silently overflow or waste context on non-OpenAI models.

Non-Goals

Full token-ID retrieval (only token counts needed in proxy hot path)
Streaming tokenization mid-response
Per-user tokenizer customization

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

Tokenizer registry / counting
Provider abstraction / adapters

Suggested naming (provisional)

Rename freely to match the change that actually lands.

Branch: feature/m2-5-g2-m1-tokenizer-registry
PR title: feat(proxy): tokenizer registry for per-model accurate token counting (m2.5.G2.M1)

What Hugging Face `tokenizers` Actually Is

It's a Rust library (with Python and no first-class Go bindings) that implements the exact same BPE/WordPiece/Unigram tokenization algorithms used to train and run open-weight models — the key property is it loads a model's actual tokenizer.json (published alongside every Hugging Face model) and reproduces token counts exactly as that model sees them, rather than approximating with a different vocabulary.

Options under consideration

Ranked starting preferences only — reopen if research or constraints point elsewhere.

The proxy is Go; tokenizers is Rust/Python. Three implementation paths (ranked by accuracy-first for production):

Run a tiny tokenizer microservice in Python (FastAPI + tokenizers library), analogous to the embedding service — POST /tokenize {model, text} -> {token_count} / {token_ids}. Simple, consistent with the existing Python-service pattern for Phase 3, adds one more network hop.
Use tokenizers' Rust core via CGo bindings (community Go bindings, e.g. github.com/daulet/tokenizers, wrap the Rust library through FFI) — keeps counting in-process in the Go proxy and avoids the network hop, but adds a CGo build dependency (cross-compilation is harder; the Rust shared library should be available at build/deploy time).
Approximate in pure Go using a rough heuristic (chars/4, or a bundled BPE re-implementation) — usually not preferred when token-budget accuracy matters. If used for early prototyping, treat it as interim until ground-truth vectors show acceptable error bounds.

Initial preference: dual-path. Use CGo in the proxy for pre-flight checks (to stay within the <20ms proxy overhead budget) and use the Python tokenizer service for Phase 3's context-assembly engine (already Python, already calling services over the network). If minimizing CGo build complexity matters, start with only the Python service and revisit the CGo path after deployment constraints are clearer.

Working notes

Preferred starting points and open questions — situational, and expected to evolve with further research during implementation.

Load once at startup

Load each tokenizer once at startup, keyed by TokenizerFamily from the capability registry (milestone 2.5.G1.M2). Cache the tokenizer.json files locally (bundled in the image or fetched once at build time) so token counting does not depend on network access to Hugging Face at runtime, which matters for air-gapped/self-hosted deployments.

Tokenizer interface

// Illustrative — exact path may differ
type Tokenizer interface {
 // Count returns the number of tokens in text for the given model.
 // Returns an error if the model is unknown or the tokenizer fails.
 Count(ctx context.Context, model, text string) (int, error)
}
 
// Registry maps TokenizerFamily -> Tokenizer implementation.
type Registry struct { ... }
 
func (r *Registry) For(family string) (Tokenizer, error)

Ground-truth verification

For every tokenizer family, include a test vector table:

// hf_cgo_test.go
var countTests = []struct {
 family string
 text string
 want int
}{
 {"llama3", "Hello world", 3},
 // ...
}

Vectors sourced from running the actual model's tokenizer offline and committing the expected counts.

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

Token counting is selectable by tokenizer family (not a single OpenAI vocabulary for every model)
OpenAI families (o200k_base, cl100k_base) match offline ground-truth vectors
Claude family uses a documented heuristic estimate backend (IsEstimate() == true)
Open-weight HF families (llama3, qwen2) — deferred (not in this milestone; requires CGo or tokenizer-service)
Runtime does not depend on live Hugging Face downloads for counting
Missing / unknown family fails clearly rather than silently approximating
Tests cover the counting path(s) that land
Repo guards / CI checks still pass

Prerequisites

2.5.G1.M2 (model capability registry) — TokenizerFamily field provides the registry key