Most RAG tutorials end exactly where production begins. You split your documents into 512-token chunks, push them through a single embedding model, and call a vector database with simple top-k retrieval. In a demo, this looks convincing. Ask the bot about your company leave policy and it returns a coherent paragraph. Everyone nods. Unfortunately, demos lie.

Production exposes every shortcut. Fixed chunks slice through legal contracts in the middle of indemnification clauses. API documentation turns into overlapping noise that drowns the signal you actually need. Latency creeps up until users abandon the query before the answer arrives. We hit this wall and had to rebuild. Our retrieval layer evolved from “semantic search and hope” into a measured, instrumented pipeline. The result was 95% recall and a 40% cut in latency. Here is what actually worked.

Match the Chunking Strategy to the Document

The 512-token default persists because it is easy, not because it is correct. Different documents carry meaning differently, and your chunking strategy should reflect that.

For legal contracts, use recursive chunking that respects structural boundaries. Legal language is nested. A clause depends on the section above it, and a fixed cut mid-sentence destroys the logic of an obligation. Recursive chunking attempts splits on natural separators first—paragraphs, then sentences—before imposing a token limit. This keeps indemnification or liability clauses intact.

For API documentation, use function-aware chunking. Developers do not search for random paragraphs; they search for endpoints, parameters, and error signatures. A chunk should contain the full function signature, its description, and the return schema as one logical unit. If you split that block in half, the retrieval system returns half the context and the generation model hallucinates the rest.

For support tickets, rely on semantic chunking that follows conversation turns. Support threads are linear and repetitive. A customer repeats the problem, an agent asks for logs, the customer attaches them. Each turn is its own semantic unit. Chunking by turns preserves who said what and when, which matters when the user asks, “What did the agent suggest on Tuesday?”

For internal wikis, try agentic chunking. Hand an LLM a section and ask it to decide where one topic ends and another begins. This costs more at ingest time, but wikis are messy. Pages contain unrelated updates from different teams, and a human-defined boundary rarely helps. Letting a model draw boundaries based on topic shifts cuts noise dramatically.

Running multiple strategies in one pipeline requires tagging documents by type at ingest. That small bit of schema discipline pays off immediately.

Combine Search Methods, Don’t Choose One

Vector search understands intent, but it routinely fails on exact matches. Ask for error code ERR_CONNECTION_REFUSED or a specific SKU, and dense embeddings often return conceptually similar but factually wrong results. BM25, the classic keyword sparse retrieval method, handles exact strings beautifully but misses semantic nuance. You need both.

Use hybrid retrieval. Run vector search and BM25 in parallel. Then combine them with Reciprocal Rank Fusion (RRF). RRF rewards documents that both methods agree are relevant, while still surfacing strong candidates from either approach. The math is simple and the result is stable: no single retrieval method dominates the final ranking.

After fusion, add a cross-encoder reranker. The first stage — vector plus sparse retrieval — is fast and broad. The cross-encoder then scores each query-document pair with full attention, meaning it actually reads the candidate against the original question. Yes, this adds latency. In our case, roughly fifty to one hundred milliseconds. But the gain in precision is sharp enough that the trade is obvious. You cannot afford to skip this if you care about recall.

Fix the Query Before You Fix the Index

Users do not write queries for your search engine. They write them for humans. “It doesn’t work” is a common support query. A vague feature description is a common internal wiki search. If you search the index with that raw input, you get garbage back.

Transform the query before it hits the retriever.

Use query expansion to generate multiple versions of the user’s question. If someone types “server down,” your system should also search for “service unavailable,” “502 error,” and “connection timeout.” Covering these intent variants moved our recall from 78% to 96%. It is a single step, and it costs almost nothing compared to the gain.

Use query decomposition for complex questions. When a user asks something like “How do I migrate from the legacy billing API to the new one and what breaking changes affect enterprise accounts?,” break it into sub-questions. One sub-question targets migration steps. Another targets enterprise-specific breaking changes. Each hits a different part of the index. The downstream language model synthesizes the final answer from well-retrieved chunks rather than guessing across a noisy context window.

Stop Guessing Hyperparameters

Once you have multiple chunking strategies, hybrid retrieval, and query transformation, you face a combinatorial problem. Chunk size, overlap, fusion weights, reranker depth, and expansion count all interact. Tweaking one in isolation breaks another. Grid search across this space is wasteful and slow.

Use Bayesian optimization instead. Treat this like a machine learning tuning job. Define your objective clearly: maximize recall while keeping latency under a ceiling. Build a golden dataset — a few hundred representative questions where you know precisely which chunks should be retrieved. Then let the Bayesian search explore the configuration space efficiently. It builds a probabilistic model of what works and tests the most promising regions next.

Every candidate configuration must pass the golden dataset before it reaches staging. If a new chunk size drops recall or a heavier reranker pushes you past the latency budget, the optimization catches it automatically. This removes opinion from the room. You stop debating whether 256 or 512 tokens is “better” and start reading the results.

The Outcome

The pipeline changes compounded exactly as we hoped.

  • Recall@10 climbed from 78% to 95%.
  • P95 latency dropped from 850 ms to 320 ms.
  • Hallucination rate fell from 12% to 3%.
  • Cost per query dropped by 38%, largely because better retrieval let us use a smaller generation model and fewer prompt tokens.

The latency reduction surprised some people on the team. Adding rerankers and query expansion sounds like it should slow things down. But because retrieval quality improved, the generation model needed less prompting, less speculation, and fewer retries. Good retrieval makes everything downstream cheaper.

Treat Retrieval Like Infrastructure

Retrieval is not a notebook you run once and forget. It is infrastructure, and it should be managed like code. Version your chunking strategies. When the legal team releases a new contract template, test your recursive splitter before it reaches production. Maintain your golden dataset as living documents, not a static CSV from last quarter. Automate your evaluations in CI so that a pull request modifying an embedding model or a fusion weight gets a comment with recall and latency numbers before a human ever reviews it.

Your users will never ask which embedding model you run. They will not care about your chunking heuristic or your reranker architecture. They care if the answer is correct, if it arrives fast, and if they can trust it. Build a pipeline that earns that trust, measure it honestly, and stop treating retrieval like an afterthought.

Source: Optimizing RAG At Scale
Join the discussion: GyaanSetu AI Community