Milestone 4.A.5 — Provider Credential Management

Status: Planned
Goal: Track A — Management API Server
Phase: 4 — Operator Platform & Multi-Provider
Estimated effort: 3 days
Track: Track A — Management API Server
ADR required: ADR-0046 — Provider credential storage and proxy retrieval path

Why This Milestone Exists

The original design hard-coded provider API keys as process-level environment variables (OPENAI_API_KEY). That's fine for a single-tenant Phase 2 deployment but breaks completely once multiple orgs need their own OpenAI/Anthropic accounts (BYO-key enterprise customers, per-org billing isolation, per-org rate limits from the vendor's own side). This milestone makes provider credentials an org-scoped, encrypted-at-rest resource — required before Track C's multi-provider routing can be multi-tenant.

Non-Goals

• Multiple keys per provider per org in v1 (deferred — one credential set per provider per org is sufficient)
• Credential rotation reminders/expiry alerts (Phase 4.5)
• AWS Bedrock IAM-role-based auth (deferred — only key-based providers in v1)

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

• open the live tree and follow existing patterns before inventing new ones
• research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
• advance the design beyond the sketch where measurement or code reality says so
• land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

• Management API
• Auth service
• Database schema / migrations

Suggested naming (provisional)

Rename freely to match the change that actually lands.

• Branch: feature/m4-a-5-provider-credentials
• PR title: feat(api): org-scoped provider credential management with envelope encryption (m4.A.5)

Schema Addition

CREATE TABLE ibex_core.provider_credentials (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 org_id UUID NOT NULL REFERENCES ibex_core.organizations(id),
 provider_name TEXT NOT NULL CHECK (provider_name IN ('openai','anthropic','azure_openai','bedrock','vllm_self_hosted')),
 -- Encrypted with the same envelope-encryption scheme as packages/crypto (Phase 1), never plaintext at rest
 encrypted_api_key BYTEA NOT NULL,
 encryption_key_id TEXT NOT NULL, -- KMS key version, enables rotation without re-encrypting rows in place
 base_url TEXT, -- for azure_openai (deployment endpoint) and vllm_self_hosted
 is_default BOOLEAN NOT NULL DEFAULT false,
 status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','disabled','invalid')),
 last_validated_at TIMESTAMPTZ,
 created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
 UNIQUE (org_id, provider_name) -- one credential set per provider per org (v1)
);
ALTER TABLE ibex_core.provider_credentials ENABLE ROW LEVEL SECURITY;
CREATE POLICY org_isolation ON ibex_core.provider_credentials
 USING (org_id = current_setting('app.org_id')::uuid);

Endpoints (illustrative)

Route shapes below are a planning sketch — names, nesting, and payloads may change during implementation.

GET /v1/organizations/{org_id}/providers → list (encrypted_api_key NEVER returned, only last-4 + status)
POST /v1/organizations/{org_id}/providers → create/update credential; validates key against provider before storing
DELETE /v1/organizations/{org_id}/providers/{provider_name} → 204; falls back to platform-level default key
POST /v1/organizations/{org_id}/providers/{provider_name}/validate → re-runs live validation, updates last_validated_at

Design notes

1. Validate-before-store: on POST, the API server makes one real (cheap, e.g. GET /v1/models or a 1-token completion) call to the provider with the submitted key before persisting it, returning 422 INVALID_CREDENTIAL immediately rather than discovering a bad key on the next customer's chat request.

2. Envelope encryption via the existing packages/crypto scheme, not a new one — reuse whatever KMS/AES-GCM pattern is already established for other secrets in the Go auth service, ported to a Python equivalent using the same key-management approach so there's exactly one KMS integration to operate, not two.

3. Proxy reads credentials via gRPC, not direct DB — add AuthService.GetProviderCredential(org_id, provider_name) → decrypted_key (auth service already owns crypto/KMS access; the proxy should not gain a new decryption dependency). This mirrors the existing "proxy calls auth via gRPC, never queries Postgres for identity" rule extended to credentials.

4. Platform-default fallback: if an org has no credential row for a provider, the proxy falls back to the platform's own key (today's OPENAI_API_KEY env var) — this is what makes the SaaS/managed deployment model work without every customer needing their own OpenAI account, while still supporting BYO-key enterprise/self-hosted deployments.

New gRPC RPC

// Added to auth service proto
rpc GetProviderCredential(GetProviderCredentialRequest) returns (GetProviderCredentialResponse);
 
message GetProviderCredentialRequest {
 string org_id = 1;
 string provider_name = 2;
}
 
message GetProviderCredentialResponse {
 string api_key = 1; // decrypted — in-process only, never logged
 string base_url = 2; // empty string if not set
 bool is_platform_default = 3; // true if falling back to platform key
}

ADR-0046 — Provider Credential Storage and Proxy Retrieval Path

Document:

• Why gRPC-mediated decryption (not proxy-side KMS access) — single point of KMS integration and audit logging.
• Why validate-before-store — fail fast on misconfiguration rather than at customer request time.
• Why one-credential-per-provider-per-org in v1 — multi-key/failover is a real future need but adds complexity not justified until a customer asks.
• Explicit non-goal: credential rotation reminders/expiry alerts (defer to Phase 4.5 notifications).

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

• Submitting an invalid key returns 422 INVALID_CREDENTIAL without persisting anything
• encrypted_api_key never appears in any API response, log line, or trace (grep-based CI check on response schemas)
• Proxy successfully resolves org-scoped credential via new GetProviderCredential RPC in an integration test
• Proxy falls back to platform default when no org credential row exists (integration test)
• Cross-tenant isolation test: org A cannot read/validate/delete org B's credentials (returns 404)

Prerequisites

• M4.A.1 skeleton merged
• packages/crypto AES-256-GCM scheme available