Provider routing and resilience

A model name is not a capability contract. The proxy resolves the requested model through tenant policy, the provider registry, and a capability manifest before it calls an upstream. On failure it may fail fast, or substitute an explicitly configured fallback model. Those are separate decisions with separate failure semantics.

Routing order

request model
  → if empty, agent.default_model
  → org model policy (deny veto)
  → registry lookup
  → adapter + credential selection
  → capability/token admission
  → upstream Complete

Model selection

1. Use a non-empty trimmed model from the request body.
2. Otherwise use the verified agent's agents.default_model from PostgreSQL.
3. Otherwise return 400 model is required.

default_provider is not used for registry lookup. The model ID remains the routing key.

Org model policy

OrgAwareRegistry wraps the process registry without changing its exact-match behavior. Policies are evaluated in ascending priority; the first matching pattern wins.

Patterns use Go filepath.Match syntax: *, ?, character classes, ranges, and negation. Two operator traps matter:

• * does not cross /. A catch-all for catalog IDs such as openai/gpt-4 needs */* or provider/*.
• The management API rejects malformed patterns that the proxy would otherwise fail closed on.

Policy reads use a process-local LRU with a 60-second TTL. Management publishes model_policy_updates:{org_id} invalidation events; a failed publish can leave a replica serving a stale decision until the TTL expires. The hot path does not fail open when policy cannot be loaded.

Capability manifest

The provider capability manifest is a preflight hint, not an execution guarantee. It records provider, model revision, endpoint identity, context limit, tokenizer family, streaming/tool/structured-output support, modalities, usage reporting, retry semantics, residency, and pricing provenance.

Missing, stale, unknown, or contradictory capability returns a typed UNSUPPORTED_CAPABILITY or NO_ROUTE error rather than silently degrading the request. Model aliases such as fast or coding must resolve to immutable deployment identities before execution.

The current manifest is intentionally limited to repository-allowlisted models. Provider catalogs and prices change independently of IBEX; adding a model is a code, capability, and test change, not a configuration-only promotion.

Circuit breakers

Breakers are independent per provider and per proxy replica. State is in-process; a shared Redis breaker is not implemented.

Client faults with a 4xx status other than 429 do not count as breaker failures. 429, 5xx, timeouts, and transport errors do. An open breaker returns 503 with Retry-After set to the cooldown.

Requests using organization BYO credentials (APIKeyOverride or BaseURLOverride) bypass the shared provider breaker. A customer's bad upstream must not open, or be blocked by, the platform breaker.

Fallback routing

Fallback is opt-in per org model policy and only for eligible failures:

Configuration lives on the first matching allow policy:

• fallback_chain is a text array with at most eight entries.
• An empty chain disables fallback.
• Runtime hops are min(chain length, IBEX_PROVIDER_FALLBACK_MAX_DEPTH); the default depth is 1.
• Every hop must pass the same org policy. A denied or unknown model is skipped; policy infrastructure failure aborts fallback and returns the original primary error.
• Fallback is only considered before any client bytes are sent. A mid-stream provider failure is not retried with a different model.

On successful substitution the proxy sets:

X-IBEX-Provider-Fallback: <reason>
X-IBEX-Provider-Used: <fallback provider>

The ClickHouse trace rows carry original_model, fallback_model, and fallback_reason; usage is attributed to the model that actually served the request. Non-fallback rows leave those fields empty.

Streaming and Responses

Chat Completions supports streaming. Idempotency keys are accepted only for non-streaming chat; a request with both streaming and an idempotency key is rejected rather than silently weakening replay semantics.

POST /v1/responses is a separate, capability-gated, non-streaming contract. It preserves ordered typed output items and unknown item/event data instead of flattening Responses into Chat Completions. Semantic SSE, replay persistence, and multi-call execution are not part of the current slice.

Failure summary

Known limits

• Breaker state is per replica; load balancing can make open/closed decisions differ across instances.
• Fallback is a policy feature, not a general retry mechanism; non-idempotent tool effects are never blindly replayed.
• The capability manifest does not prove provider account entitlement, current model availability, or production residency.
• No hosted provider-latency or failover SLO is published.

Related

• Provider adapters — supported wire dialects and mock/live behavior
• Request routing — middleware order and model handoff
• Proxy configuration — environment variables
• Responses — non-streaming contract
• Status and evidence — what is not proven