Overview
High-level system diagram, design philosophy, and honest Phase 2 shipped scope for IBEX Harness.
IBEX Harness is a distributed platform for an authenticated LLM proxy today, with persistent agent memory and context assembly planned for Phase 3. Agent applications call a low-latency LLM proxy; the proxy authenticates every request through the auth service, applies org directives and session context, then forwards to a mock or live provider.
System diagram
The critical path is every LLM request: authenticate, enforce limits, resolve directive/session, call the provider, stream the response. Target proxy overhead is under 20ms (p99) excluding provider latency. Trace emission and future memory extraction must never block inference.
Design principles
Performance first
The proxy path is optimized for millisecond budgets. Auth validation has a 50ms gRPC deadline; auth cache and Redis timeouts keep the hot path bounded.
Security by default
org_id comes from the verified token, never the request body. Postgres RLS, Redis key namespacing, and permission bitmaps enforce isolation at every layer. Cross-tenant misses return 403, not 404.
Fail gracefully
Auth unreachable → fail closed (503). Redis down → rate limit fail-open; auth cache skips wrap. ClickHouse down → traces drop, chat continues.
Observable everything
Structured JSON logs with request_id, Prometheus metrics on bounded labels, OpenTelemetry traces, and optional ClickHouse llm_traces batches.
What runs synchronously vs async
Synchronous (blocks the agent)
Token validation, agent identity check, rate limiting, directive resolve, chat parse, provider call, and response streaming.
Asynchronous (never blocks)
ClickHouse trace emission, session checkpoint writes, and (Phase 3+) memory extraction, drift alerts, and billing counters.
Today
Mock or live provider forwarding is live. Memory/context assembly remain Phase 3 — do not design clients against those APIs yet.
Latency budgets
| Operation | Budget |
|---|---|
Auth ValidateToken gRPC | 50ms |
| Redis rate limit check | 5ms |
| Full proxy overhead (excl. LLM) | 20ms p99 |
| Context assembly (Phase 3) | 50ms p95 (planned) |
Related
- Services — which components are live vs planned
- Request lifecycle — step-by-step proxy flow
- Glossary — PAT, RLS, org_id, and other terms
Was this page helpful?
Last updated on