Most RAG prototypes look the same under the hood. Someone feeds a PDF into a pipeline, slices the text into neat 512-token chunks, dumps them into a vector database, and calls the job done. For a hallway demo, this can look impressive. In production, it collapses.

A fixed chunk does not care what it cuts. It will split a legal contract in the middle of an indemnity clause. It will stuff five unrelated API endpoints into the same context window and drown the model in noise. It will force you to retrieve more fragments than necessary, bloating latency and burning tokens. The result is half-answers, hallucinations, and frustrated users.

We tore our retrieval layer down to the studs and rebuilt it. The outcome was a system that hit 95 percent recall while cutting latency by 40 percent. Here is exactly how we did it.

Why Fixed Chunks Die in Production

The 512-token default is not a design choice. It is a by-product of early embedding model context windows and tidy library defaults. It is easy to implement and catastrophic to rely on.

Documents are not uniform. A legal clause can run seven hundred tokens without a clean break. Slice it at five hundred and twelve, and you create two orphaned fragments. When a lawyer or compliance officer asks about liability caps, the system returns half the obligation. The language model hallucinates the missing half, or worse, denies the cap exists.

API documentation suffers from the opposite ailment. A five-hundred-token chunk might swallow an entire module: authentication headers, error codes, rate limits, and webhook schemas. When a developer asks how to handle AUTH_4027, the retriever surfaces a stew of unrelated functions. The model has no choice but to average them into generic mush.

Bad chunking also inflates latency. Weak fragments mean you need a larger top-k to cover a topic. More chunks mean longer prompts. Longer prompts mean slower generation and higher bills. The user experience dies by a thousand cuts.

Match the Chunk to the Document

We stopped counting tokens and started reading the material. The right chunking strategy depends on the structure of the source.

Legal documents need recursive character chunking with clause-aware boundaries. The splitter respects hierarchy: it looks for section headers first, then numbered paragraphs, then natural sentence breaks. It never severs a sub-clause or splits an obligational phrase across chunks. When you retrieve a passage about indemnification, you get the full clause, the cap, and the exceptions.

API documentation demands structure-aware chunking. We parse by function definition, not token budget. Each chunk contains a complete function signature, its parameter descriptions, and the immediately adjacent error handling notes. If a developer searches for a specific method, they receive the entire contract, not a fragment caught inside an arbitrary split.

Support tickets are noisy and nonlinear. A thread might start with a bug report, introduce a workaround, and end with an internal escalation note. Semantic chunking detects topic shifts by measuring embedding similarity between sentences. We allow breaks only at natural thematic boundaries, so a conversation about login failures stays separate from a follow-up about billing cycles.

Wikis were the hardest. They are sprawling, cross-linked, and loosely organized. We used agentic chunking, where a lightweight LLM reads a page and decides the breaks based on thematic coherence. It costs slightly more at ingestion time, but the resulting chunks are self-contained and retrieval-ready. A page about deployment best practices splits into logical units: pre-flight checks, rollback procedures, and monitoring setup, rather than arbitrary text blocks.

Hybrid Retrieval: Keywords and Vectors Together

Dense vector search understands meaning. It is terrible at exact strings. If a user searches for a precise error code like AUTH_4027 or a customer name like "Stark Industries," vector embeddings can miss the target because they optimize for conceptual proximity, not character-level accuracy.

Pure keyword search through BM25 has the inverse flaw. It will find AUTH_4027 perfectly, but it will miss the conceptual bridge between "authorization failure" and "login denied."

Li eseguiamo entrambi in parallelo. BM25 e la ricerca vettoriale operano indipendentemente sullo stesso corpus. Le loro liste di risultati vengono unite tramite Reciprocal Rank Fusion, che riordina i candidati bilanciando i loro ranghi posizionali. Non sono necessari pesi calibrati. Si ottiene semplicemente la precisione del match esatto e l'intuizione della ricerca semantica in un'unica lista ordinata.

Poi aggiungiamo un reranker cross-encoder. Si tratta di un modello separato che assegna un punteggio a ogni passaggio rispetto alla query originale, producendo un segnale di rilevanza molto più fine rispetto a quello di un singolo retriever. Aggiunge circa 50 millisecondi di latenza. Aumenta il recall del 15 percento. Se la qualità della risposta è una priorità, questo compromesso è imprescindibile.

Query Expansion: correggi la ricerca prima che inizi

Le query errate sono il segreto sporco di ogni sistema di retrieval. Gli utenti non scrivono come il tuo spazio di embedding. Digitano "si è rotto". Incollano stack trace troncati. Usano gergo interno che il tuo indice non ha mai visto.

Trasformiamo la query prima ancora che tocchi l'indice. Per prima cosa, espandiamo una singola query in tre o cinque termini di ricerca diversi. Se l'originale è "payment failed", cerchiamo anche "transaction error", "billing declined" e "charge unsuccessful