Most RAG tutorials end at the demo. You chunk by token count, stuff everything into a vector database, and call it a day. That works when a user asks "What is the return policy?" into a clean FAQ. It falls apart when someone pastes half a contract and asks about clause three, or when a developer types an obscure error code into your documentation search.

Fixed token windows cut legal agreements off mid-sentence. Large chunks bury API references under paragraphs of noise. Worst of all, slow retrieval makes users abandon the query before the model even starts generating. We learned this the hard way. When we moved our retrieval layer from hope to measurement, we cut latency by forty percent and pushed recall to ninety-five percent. Here is exactly what changed.

Smart Chunking

Stop thinking about chunk size as a magic number. A 512-token window makes no sense for a legal document where a single clause spans multiple paragraphs, and it is equally useless for API docs where a function signature and its two-line description should stay together. We switched to structure-aware splitting.

For legal text, recursive chunking respects document hierarchy. Clauses stay intact. For API documentation, we use function-aware splitting that keeps signatures, parameters, and examples together as atomic units. Support tickets and conversational data need semantic boundaries, splitting where the topic shifts rather than at an arbitrary character count. The result is that each chunk carries enough context to be useful but not so much that it dilutes the signal. Your embedding model has a limited attention budget. Spend it wisely.

Hybrid Retrieval

Vector search excels at finding conceptually similar content. Ask about slow database queries and it surfaces performance tuning guides. But ask for "Error 0x80070057" and semantic search drifts into unrelated territory because dense embeddings do not handle exact matches well. BM25, on the other hand, nails exact strings and rare terms, yet it has no idea that "latency" and "slow response time" mean the same thing.

We run both in parallel and merge them with Reciprocal Rank Fusion. RRF is simple and effective. It takes the ranked lists from each method and scores documents based on their position, giving strong candidates from either system a fair shot. After fusion, we run a cross-encoder reranker over the combined results and return only the top five. The reranker adds about fifty milliseconds of latency but improved our recall by fifteen percent. That trade-off pays for itself many times over in generation quality.

Query Expansion

Users are terrible at querying. They abbreviate, misspell, or pack three questions into one long ramble. If you search exactly what they typed, you miss the documents they actually need.

We now transform every query before it hits the index. First, we generate multiple rephrased versions of the original question to cover synonyms and alternate phrasings. Second, we decompose complex questions into smaller sub-questions. A query like "Why is the refund failing for international customers and how do I fix it?" becomes two distinct searches: one about international refund failures and one about remediation steps. Query expansion alone moved our recall from seventy-eight percent to ninety-four percent. The lesson is straightforward: do not trust the user's first draft. Help them.

Stop Guessing, Start Searching

Chunk size, overlap percentage, top-k cutoff, and reranker depth interact in ways that are impossible to tune by hand. We spent too many cycles debating whether 256 tokens was better than 512 while ignoring the overlap setting that was actually destroying coherence.

We replaced intuition with Bayesian optimization. Instead of grid search, which wastes compute on obviously bad regions, Bayesian methods build a probabilistic model of what works and actively seek the Pareto frontier that maximizes recall while minimizing latency. For our stack, that meant finding the specific combination of chunk size, overlap, and top-k that gave us ninety-five percent recall without blowing our latency budget. Different use cases landed on different points along that frontier. Customer-facing chatbots prioritized speed. Internal legal research prioritized recall. Automated optimization let us serve both without manual copy-pasting of config files.

The Results

De cijfers spreken voor zich. Onze Recall@10 steeg van achtenzeventig procent naar vijfennegentig procent. De p95-latentie daalde van 850 milliseconden naar 320 milliseconden. En omdat het model eindelijk relevante context kreeg in plaats van ruis, daalde het hallucinatiepercentage van twaalf procent naar drie procent. Betere retrieval maakt antwoorden niet alleen sneller. Het maakt ze waarheidsgetrouw.

Wat nu te doen

Als je je retrieval-laag aan het herbouwen bent, begin dan hier:

  • Chunk op basis van documentstructuur, niet op tokenaantal. Stem je splitting-strategie af op de vorm van je data.
  • Gebruik hybrid retrieval. Combineer vector search met BM25, voeg ze samen met Reciprocal Rank Fusion en voer een rerank uit voordat je gaat genereren.
  • Breid queries uit voor een betere dekking. Herschrijf en ontleed ze voordat de zoekopdracht wordt uitgevoerd.
  • Bouw een golden dataset voor testen. Je kunt niet optimaliseren wat je niet meet.
  • Optimaliseer parameters met geautomatiseerde tools. Bayesian search vindt betere instellingen dan je onderbuikgevoel.

Retrieval is geen configuratiebestand dat je eenmalig instelt en vervolgens vergeet. Het is infrastructuur, en infrastructuur verdient dezelfde nauwkeurigheid als productiecode: tests, metingen en continue optimalisatie. Behandel het zo, en je RAG-systeem stopt met een demo zijn en wordt een echt product.

Optionele leercommunity: GyaanSetu AI