ADR-0022: Health check contract (Phase 1)

• Status: Accepted
• Date: 2026-06-05
• Authors: IBEX Harness team

Context

M1.4.3 requires a documented, consistent health check contract across Go services. Before this ADR:

• /health and /ready returned ad-hoc JSON (status + optional reason)
• Auth readiness used a Postgres TCP dial only (not SELECT 1)
• Proxy readiness checked Redis but not auth gRPC
• Logic was duplicated in services/*/internal/health

Kubernetes operators need predictable probe behaviour: liveness restarts unhealthy pods; readiness removes pods from load balancing without restart.

Decision

1) Endpoints

2) Response schema

{
  "status": "ok",
  "checks": {
    "postgres": { "status": "ok", "latency_ms": 2 },
    "redis": { "status": "failed", "message": "connection refused", "latency_ms": 501 }
  }
}

• status (top-level): ok | degraded | unhealthy
• checks[name].status: ok | failed
• checks[name].message: present when failed (no secrets)
• checks[name].latency_ms: wall time for that checker

/health body: {"status":"ok","checks":{}} always.

3) Readiness HTTP mapping

4) Checker classification (Phase 1)

Auth — critical:

• postgres: SELECT 1 via *sql.DB
• grpc: TCP reachability to the gRPC listen address

Proxy — critical:

• auth_grpc: ValidateToken with sentinel probe token ibex_health_probe_invalid; codes.Unauthenticated means reachable
• redis: PING over RESP

Advisory (Phase 1): none. Phase 2 may add LLM provider reachability as advisory (degraded but still HTTP 200).

5) Timeouts

• 500ms per checker (PerCheckTimeout)
• 750ms overall request budget (OverallTimeout)
• Checkers run in parallel within the overall budget

6) packages/healthcheck

Shared implementation used by auth and proxy. Services register checkers in main; HTTP routers delegate to Server.HealthHandler() / ReadyHandler().

Consequences

• Breaking change: not_ready + reason replaced by checks map (documented here)
• make dev-smoke unchanged (HTTP status codes only)
• K8s probe examples live in OPS_GUIDE.md

References

• M1.4.3 milestone
• ADR-0020