---
title: "Production architecture, observability, cost, and resilience"
chapter: "22"
---

# Production architecture, observability, cost, and resilience

A production AI request crosses API gateway, identity, policy, orchestration,
retrieval/tools, provider, validation, tracing, and a user experience that can
recover.

## Reference flow

`client → auth/rate limit → AI use case service → policy/context builder →
retrieval/tool gateway → model gateway → output validator → response/audit`

Keep provider clients behind role-based gateways. Keep domain actions behind
normal application services. Persist explicit workflow state for long jobs.

## Observability

Record trace/request IDs, release, tenant class, role/model, prompt version,
retrieval IDs/scores, tool names/status, latency phases, tokens, estimated cost,
finish reason, schema/retry/abstention, and user outcome. Redact content.

Spring AI uses Micrometer observations for model and vector operations.
Sensitive prompt and vector response logging is disabled by default for a good
reason.

## Resilience

Use bulkheads per provider/role, bounded queues, timeouts, circuit breakers,
idempotency, load shedding, and explicit degraded behavior. Retry transient
reads with jitter; never replay uncertain writes blindly.

## Cost

Budget input/output tokens, embeddings, reranking, tool/API calls, retrieval,
storage, egress, and local hardware. Cache only when identity, permissions,
freshness, and model/prompt version make reuse safe. Semantic caches need the
same security boundaries as search.

## Deployment

Ship an immutable application with versioned prompts, evaluation report, model
registry, SBOM, migrations, and rollback. Canary by role/use case and monitor
quality—not only HTTP errors.

## Feynman check

The model is one engine in a factory. Production engineering controls the
conveyor belts, safety guards, meters, maintenance, and emergency stop.