Real-life scenario: ClaimsPilot

ClaimsPilot helps insurance adjusters review a claim packet, find policy evidence, detect missing information, draft a recommendation, and prepare approved actions. It never makes or pays a final claim autonomously.

User outcome

An adjuster uploads forms, photos, repair estimates, emails, and a policy ID. Within two minutes the workspace should:

extract typed facts with source coordinates;
retrieve current policy clauses valid on the loss date;
identify conflicts and missing evidence;
draft coverage reasoning with claim-level citations;
call read-only tools for claim/account state;
propose—not execute—reserve or communication changes;
route uncertainty and high-risk cases to a human.

Architecture

Angular UI → Spring Boot 4 / Spring AI 2 → policy/authorization → document pipeline → pgvector + lexical search → reranker → model router → claim tools → schema/grounding checks → approval queue → audit

OpenAI GPT-5.6 Sol handles the hardest reasoned draft when approved for the data class. Gemini 3.6 Flash handles multimodal packet understanding and routine interaction. Ollama handles private preliminary extraction and an independent groundedness checker. These are evaluated roles, not automatic fallbacks.

Ingestion

Files are malware-scanned, authorized, parsed/OCRed, and split by form section, policy clause, or image/page. Every chunk stores claim/tenant, ACL, source hash, page/coordinates, loss-date applicability, parser version, embedding identity, and retention key.

Policy material enters a separate versioned corpus. An effective-date filter is mandatory. Old policy wording is retained for claims arising under it.

Retrieval

The system extracts exact policy/form identifiers for lexical retrieval and embeds the natural-language question for semantic retrieval. It fuses, reranks, and rejects low relevance. Citations are created from metadata and validated against returned evidence.

Spring AI implementation

record ClaimDraft(
  List<Fact> facts,
  List<CitedReason> reasons,
  List<MissingItem> missing,
  ProposedAction proposedAction,
  boolean humanReviewRequired) {}

ClaimDraft draft = claimsClient.prompt()
    .advisors(claimMemory, policyRag, auditAdvisor)
    .tools(new ClaimReadTools(claimService, authorization))
    .system(CLAIMS_POLICY)
    .user(packetQuestion)
    .call()
    .entity(ClaimDraft.class);

validator.validateSchema(draft);
grounding.validateEveryCitation(draft);
policyEngine.validateProposal(user, claim, draft.proposedAction());
approvalQueue.submit(draft); // no payment or customer message yet

ToolCallingAdvisor handles the bounded read-only loop. Write tools are absent. After an adjuster approves an exact proposal, a separate normal REST command performs authorization, optimistic locking, idempotency, and audit.

Failure paths

no policy clause: abstain and request a policy specialist;
conflicting OCR: show both sources and request confirmation;
stale claim version: return conflict and rebuild the proposal;
provider timeout: save resumable job state; do not claim success;
malicious instruction inside email: treat as quoted evidence, never a command;
low retrieval score: no-answer state, not a creative guess;
model/schema failure: one bounded correction, then human queue;
tool unavailable: state the missing fact and avoid inventing it.

Evaluation and rollout

The golden set includes routine, ambiguous, denied, fraud-sensitive, multilingual, scanned, conflicting, injection, and cross-tenant cases. Release gates require zero unauthorized tool/data access, complete valid citations, high policy-clause recall, bounded latency/cost, and no regression by claim slice.

Shadow evaluation uses redacted approved cases. Canary begins with read-only drafts for trained adjusters. Expansion requires measured correction rate, groundedness, time saved, escalation quality, and incident-free evidence.

Glossary and abbreviations

Term	Plain meaning
AI / ML	Artificial intelligence / machine learning
LLM / VLM	Language model / vision-language model
Token / context window	Text piece / maximum working input-output space
Inference	Running fixed model weights to produce output
RAG	Retrieve evidence, then generate with it
Embedding	Numeric coordinate representing content meaning
ANN / HNSW / IVFFlat	Approximate search / two vector index approaches
BM25 / hybrid search	Lexical ranking / lexical plus vector retrieval
Reranker	More precise second-stage candidate scorer
SFT / DPO / RFT	Supervised / preference / reinforcement fine-tuning
LoRA / QLoRA	Small trainable adapter / quantized adapter training
Quantization	Smaller numeric weights for cheaper inference
Tool calling	Model requests a typed application function
MCP	Model Context Protocol for tools, resources, and prompts
Agent	Bounded model-driven observe/action loop
Advisor	Spring AI interceptor/orchestration component
ETL	Extract, transform, load document pipeline
ACL / RBAC	Access list / role-based access control
PII	Personally identifiable information
MRR / nDCG	Retrieval ranking quality metrics
Groundedness	Claims supported by supplied evidence
Hallucination	Unsupported model-generated content
Golden set	Stable labeled evaluation examples
Shadow / canary	Compare invisibly / expose to a controlled slice
TTFT	Time to first token
KV cache	Reused transformer attention state
GPU / VRAM	Accelerator / its working memory

Official references

Final Feynman challenge

Trace one ClaimsPilot question from authenticated user, ingestion and ACL, through hybrid retrieval, embeddings, reranking, prompt assembly, model route, tool call, structured draft, citation validation, approval, command, observability, evaluation, canary, and rollback.