Evidence and relay
Evidence is written to a PostgreSQL transactional outbox and projected asynchronously by a default-off relay daemon. The outbox is the durable handoff contract; ClickHouse is a pilot projection, not the source of truth.
Write-side contract
The outbox envelope uses evidence.v1 and carries:
- Tenant and event identity:
org_id,event_id, aggregate ID, aggregate sequence - Schema and event type
- A payload plus SHA-256 digest and digest scheme
- A redaction class
- A tombstone/version fence
Known event types include run committed, span committed, assembly metrics, score candidates, directive snapshot, tool audit, and session event. The outbox is intentionally narrow: it is not a general model-policy invalidation bus or an organization-deletion cascade mechanism.
Payload and metadata are written in one PostgreSQL transaction with the durable effect. A payload cannot assign its own redaction_class or tombstone_fence; the producer owns those fields. Invalid JSON, a mismatched digest, a caller-supplied fence, or a prohibited class is rejected at write time.
Delivery protocol
States
| State | Meaning |
|---|---|
pending | Eligible for claim when available_at allows |
in_flight | Claimed by one relay cycle; not yet finalized |
delivered | Sink accepted the row |
failed | Transient failure; retry after backoff |
poison | Terminal: permanent validation/fence failure or attempt limit reached |
A stale in_flight row is returned to pending only after the recovery window. A crash after claim but before acknowledgement therefore causes redelivery, not loss. The sink must make redelivery idempotent using stable event identity.
Validation order
Before any sink call, the relay validates row identity, redaction class, tombstone fence, payload presence, digest format, and digest match. Only the metadata redaction class is currently projectable. redacted is not projectable in this slice; unclassified, prohibited, and unsupported classes fail closed.
A fence must name a resource type, resource ID, and positive version. A non-session fence or malformed fence is permanent; an ordinary database error while checking the fence is retryable.
Fence semantics
| Result | Action |
|---|---|
alive | Deliver through the fence-aware sink |
tombstoned | Poison; never resurrect deleted data |
stale_version | Poison; the row references an obsolete resource version |
not_found | Poison |
| Database error | Leave retryable |
The replay check and the sink write are not one database transaction. This is the central recovery gap: the design narrows the race with an authoritative fence and idempotent sinks, but does not prove an atomic fence-and-project boundary.
Retry and dead-letter behavior
Transient failures use exponential backoff with cryptographic jitter, capped at 60 seconds. Permanent delivery failures poison immediately. A row also poisons when its post-claim attempt count reaches IBEX_RELAY_MAX_ATTEMPTS.
Permanent classes include unclassified/redacted/prohibited redaction, missing or invalid fence, fenced delivery being unavailable, unsupported fence resource, and a tombstoned resource. A poison row is retained for inspection; it is not silently dropped.
Process and identity
| Behavior | Detail |
|---|---|
| Disabled | Validates configuration, logs a reason, exits 0 without connecting |
Missing IBEX_ENV | Fails closed, exit 1 |
| Enabled without both DSNs | Fatal startup error |
| Shutdown | Stops the loop, waits for the in-flight batch, closes HTTP and dependencies |
| Abandoned claims | Recovered by stale recovery, not released at shutdown |
The daemon connects as a login role but pins ibex_evidence_relay per connection and asserts current_user at startup. ibex_evidence_relay is NOLOGIN, is not granted to ibex_app, and receives only the helper privileges required by migration 000047. The HTTP surface is /health, /ready, and /metrics; there is no relay operator API.
Configuration
| Variable | Default | Bounds |
|---|---|---|
IBEX_RELAY_ENABLED | false | Boolean |
IBEX_RELAY_POSTGRES_DSN | — | Required when enabled; secret |
IBEX_RELAY_CLICKHOUSE_DSN | — | Required when enabled; secret |
IBEX_RELAY_POLL_INTERVAL | 2s | 500ms–5m |
IBEX_RELAY_BATCH_SIZE | 32 | 1–256 |
IBEX_RELAY_MAX_ATTEMPTS | 8 | 1–100 |
IBEX_RELAY_RECOVER_AFTER | 60s | 1s–1h |
IBEX_RELAY_OPERATION_TIMEOUT | 30s | 1s–5m |
IBEX_RELAY_ERROR_BACKOFF | 5s | 500ms–1m; doubles, cap 1m |
IBEX_SHUTDOWN_TIMEOUT | 30s | Shared graceful-shutdown budget |
A missing or unreachable dependency when enabled is a fatal startup error. Secrets are never logged, and error messages do not echo DSNs.
Observability
ibex_relay_batches_total{result}, claimed, delivered, retried, dead-letter, and recovered counters; batch, delivery, and fence-check duration histograms; and ibex_process_up. Labels are bounded, and unknown values collapse to error.
Alert on a rising dead-letter count, stale claims that never recover, repeated database errors, and a relay that is enabled but has no successful batches. A high delivered count does not by itself prove end-to-end durability before chat acknowledgement.
Known limitations
- ClickHouse remains a non-production projection pilot; there is no production sink.
- No hosted or HA recovery evidence exists.
- No relay-level tenant/agent allowlist, request-auth layer, or operator endpoint.
- The fence read, outbox write, and projection are not one atomic transaction.
- Chat acknowledgement is not response-bound, so durable-before-ack is not proven end to end.
- GAP-005 and GAP-011 remain open.
Verify
go test ./packages/evidenceoutbox/...
go test ./services/relay/...
POSTGRES_TEST_DSN=postgres://ibex:ibex@127.0.0.1:5433/ibex_test?sslmode=disable \
go test -tags=integration -p 1 ./services/relay/...
Use -p 1 for integration tests so migration locks do not collide. Local tests cover claim, poison, tombstone fence, stale recovery, role denial, and redaction rejection; they do not constitute a production recovery drill.
Related
- Data lifecycle and deletion — tombstone semantics
- Traces and ClickHouse — analytics versus canonical traces
- Failure modes and recovery — restore classes
- Billing and budgets — separate ledger authority
- Tenant isolation — org predicates and RLS boundaries