Production architecture, observability, cost, and resilience

A production AI request crosses API gateway, identity, policy, orchestration, retrieval/tools, provider, validation, tracing, and a user experience that can recover.

Reference flow

client → auth/rate limit → AI use case service → policy/context builder → retrieval/tool gateway → model gateway → output validator → response/audit

Keep provider clients behind role-based gateways. Keep domain actions behind normal application services. Persist explicit workflow state for long jobs.

Observability

Record trace/request IDs, release, tenant class, role/model, prompt version, retrieval IDs/scores, tool names/status, latency phases, tokens, estimated cost, finish reason, schema/retry/abstention, and user outcome. Redact content.

Spring AI uses Micrometer observations for model and vector operations. Sensitive prompt and vector response logging is disabled by default for a good reason.

Resilience

Use bulkheads per provider/role, bounded queues, timeouts, circuit breakers, idempotency, load shedding, and explicit degraded behavior. Retry transient reads with jitter; never replay uncertain writes blindly.

Cost

Budget input/output tokens, embeddings, reranking, tool/API calls, retrieval, storage, egress, and local hardware. Cache only when identity, permissions, freshness, and model/prompt version make reuse safe. Semantic caches need the same security boundaries as search.

Deployment

Ship an immutable application with versioned prompts, evaluation report, model registry, SBOM, migrations, and rollback. Canary by role/use case and monitor quality—not only HTTP errors.

Feynman check

The model is one engine in a factory. Production engineering controls the conveyor belts, safety guards, meters, maintenance, and emergency stop.