---
title: "Documents, ETL, parsing, chunking, and provenance"
chapter: "11"
---

# Documents, ETL, parsing, chunking, and provenance

Retrieval quality begins before embeddings. Spring AI's ETL model uses
`DocumentReader → DocumentTransformer → DocumentWriter`.

## Ingestion stages

`discover → authorize → fetch → parse → normalize → structure → chunk →
enrich metadata → embed → index → verify → publish`

Store a source manifest with content hash, parser/chunker/embedder versions,
tenant, ACL, effective dates, ingestion run, and deletion key.

## Parsing

PDFs, HTML, Office files, tickets, tables, and scanned images need different
parsers. Preserve headings, page numbers, table boundaries, links, and source
IDs. OCR confidence should influence retrieval or review.

## Chunking

Chunk by semantic structure first, then token limits. Use overlap only where
meaning crosses boundaries; too much overlap produces duplicate evidence.
Parent-child retrieval can index small passages and return a larger parent.

Evaluate chunk size with real questions. Small chunks improve precision but
lose context; large chunks dilute relevance and consume model tokens.

## Idempotency and deletion

Use deterministic chunk IDs from source version and location. Reprocessing the
same version must not duplicate vectors. A source delete or permission change
must remove or quarantine every derived chunk promptly.

## Incremental ingestion

Read change feeds or timestamps, but periodically reconcile against the source
of truth. Handle partial failure with resumable stages and a dead-letter queue.
Do not publish half an updated document set as current.

## Feynman check

RAG is cooking with library pages. Bad scanning, random cuts, missing labels,
or stale ingredients cannot be repaired by a better chef.