Documents, ETL, parsing, chunking, and provenance

Retrieval quality begins before embeddings. Spring AI's ETL model uses DocumentReader → DocumentTransformer → DocumentWriter.

Ingestion stages

discover → authorize → fetch → parse → normalize → structure → chunk → enrich metadata → embed → index → verify → publish

Store a source manifest with content hash, parser/chunker/embedder versions, tenant, ACL, effective dates, ingestion run, and deletion key.

Parsing

PDFs, HTML, Office files, tickets, tables, and scanned images need different parsers. Preserve headings, page numbers, table boundaries, links, and source IDs. OCR confidence should influence retrieval or review.

Chunking

Chunk by semantic structure first, then token limits. Use overlap only where meaning crosses boundaries; too much overlap produces duplicate evidence. Parent-child retrieval can index small passages and return a larger parent.

Evaluate chunk size with real questions. Small chunks improve precision but lose context; large chunks dilute relevance and consume model tokens.

Idempotency and deletion

Use deterministic chunk IDs from source version and location. Reprocessing the same version must not duplicate vectors. A source delete or permission change must remove or quarantine every derived chunk promptly.

Incremental ingestion

Read change feeds or timestamps, but periodically reconcile against the source of truth. Handle partial failure with resumable stages and a dead-letter queue. Do not publish half an updated document set as current.

Feynman check

RAG is cooking with library pages. Bad scanning, random cuts, missing labels, or stale ingredients cannot be repaired by a better chef.