ADR-0025: LLM provider abstraction

Status: Accepted
Date: 2026-07-12
Authors: IBEX Harness team
Milestone: 2.1.1 Provider interface and registry

Context

When this ADR was written, Phase 1 validated and authenticated every POST /v1/chat/completions request but returned 501 PROVIDER_NOT_CONFIGURED with no upstream LLM call. Phase 2 later added mock/live forwarding via packages/provider. Anthropic lands in Phase 2.5 (ADR-0040); Azure OpenAI and AWS Bedrock remain later multi-provider work. Hard-coding OpenAI HTTP calls in the proxy handler would require a large refactor when multi-provider support lands.

The proxy already parses OpenAI-shaped JSON in services/proxy/internal/llm (Phase 1). Provider communication needs a shared abstraction in packages/provider with no service imports.

Decision

1) Single `Provider` interface with one `Complete` method

type Provider interface {
    Complete(ctx context.Context, req Request) (Response, error)
    Name() string
    SupportedModels() []string
}

Request.Stream bool selects streaming vs non-streaming inside the implementation. A split interface (Streamer + Completer) would force callers to type-assert on every request.

2) Model routing and capabilities live in `Registry`, not providers

Each provider declares SupportedModels(). Registry.For(model) returns the implementation. Phase 2.5 registers Anthropic (ADR-0040); later phases add Azure OpenAI / Bedrock without changing handlers or middleware contracts.

NewRegistry takes a CapabilityCatalog and returns ErrDuplicateModel / ErrMissingCapability on conflicts or missing metadata — fail fast at startup, not on first customer request. Registry.Capability(model) exposes curated context window / feature / tokenizer-family rows (ADR-0041).

3) API keys are constructor details, not interface fields

Provider credentials come from environment variables (Phase 2) or org-scoped config (later). The Provider interface is key-agnostic; packages/provider/openai/ reads keys in its constructor.

4) `Response.Body` is `io.ReadCloser`

Streaming responses must not be fully buffered on the hot path. Non-streaming callers read the full body and decode JSON. The caller closes Body.

5) Messages-only request contract; directive injection in proxy

provider.Request.Messages is authoritative when Complete is called. Agent directive injection (milestone 2.3.3) runs in proxy middleware (packages/injection) and mutates Messages before the provider call. Provider implementations must not implement injection logic — this avoids duplicating injection between OpenAI client (2.1.2) and proxy middleware (2.3.3).

6) Package boundary

packages/provider imports only stdlib and other packages/* modules. No services/* imports. Concrete implementations live in subpackages (e.g. packages/provider/openai/ in milestone 2.1.2).

7) Registry lifecycle

Built once at proxy startup via provider.NewRegistry(catalog, ...). Read-only after construction. Milestone 2.1.1 shipped an empty registry (no providers registered); 2.1.2 registers OpenAI; 2.5.G1.M2 requires a capability entry for every registered model ID.

8) Error types

ErrNoProviderForModel → proxy returns 501 PROVIDER_NOT_CONFIGURED
ErrMissingCapability / ErrDuplicateModel → startup/config failure (never silent at request time)
ProviderError carries provider HTTP status and raw body for mapping in 2.1.5 — never log ProviderBody

9) Phase 2 proxy Postgres exception

Phase 1 rule: proxy has no Postgres for identity (auth via gRPC only). Phase 2 milestones 2.3.2 and 2.4.x add a proxy-owned database/sql pool for directive and session stores (including session writes), opened in bootstrap only. Formal decision: ADR-0039.

Consequences

Positive:

Phase 2.5 Anthropic registration is already live; later Azure/Bedrock work stays registration-only
Shared package testable offline at ≥90% unit coverage
Handler stays provider-agnostic
Milestone 2.1.4 extracted registry lookup from handler into ProviderRoutingMiddleware (no behavior change)

Negative:

Duplicate Message types in llm and provider — conversion function in 2.1.2 bridges them

References

Was this page helpful?