Most RAG tutorials end at the demo. You chunk by token count, stuff everything into a vector database, and call it a day. That works when a user asks "What is the return policy?" into a clean FAQ. It falls apart when someone pastes half a contract and asks about clause three, or when a developer types an obscure error code into your documentation search.

Fixed token windows cut legal agreements off mid-sentence. Large chunks bury API references under paragraphs of noise. Worst of all, slow retrieval makes users abandon the query before the model even starts generating. We learned this the hard way. When we moved our retrieval layer from hope to measurement, we cut latency by forty percent and pushed recall to ninety-five percent. Here is exactly what changed.

Smart Chunking

Stop thinking about chunk size as a magic number. A 512-token window makes no sense for a legal document where a single clause spans multiple paragraphs, and it is equally useless for API docs where a function signature and its two-line description should stay together. We switched to structure-aware splitting.

For legal text, recursive chunking respects document hierarchy. Clauses stay intact. For API documentation, we use function-aware splitting that keeps signatures, parameters, and examples together as atomic units. Support tickets and conversational data need semantic boundaries, splitting where the topic shifts rather than at an arbitrary character count. The result is that each chunk carries enough context to be useful but not so much that it dilutes the signal. Your embedding model has a limited attention budget. Spend it wisely.

Hybrid Retrieval

Vector search excels at finding conceptually similar content. Ask about slow database queries and it surfaces performance tuning guides. But ask for "Error 0x80070057" and semantic search drifts into unrelated territory because dense embeddings do not handle exact matches well. BM25, on the other hand, nails exact strings and rare terms, yet it has no idea that "latency" and "slow response time" mean the same thing.

We run both in parallel and merge them with Reciprocal Rank Fusion. RRF is simple and effective. It takes the ranked lists from each method and scores documents based on their position, giving strong candidates from either system a fair shot. After fusion, we run a cross-encoder reranker over the combined results and return only the top five. The reranker adds about fifty milliseconds of latency but improved our recall by fifteen percent. That trade-off pays for itself many times over in generation quality.

Query Expansion

Users are terrible at querying. They abbreviate, misspell, or pack three questions into one long ramble. If you search exactly what they typed, you miss the documents they actually need.

We now transform every query before it hits the index. First, we generate multiple rephrased versions of the original question to cover synonyms and alternate phrasings. Second, we decompose complex questions into smaller sub-questions. A query like "Why is the refund failing for international customers and how do I fix it?" becomes two distinct searches: one about international refund failures and one about remediation steps. Query expansion alone moved our recall from seventy-eight percent to ninety-four percent. The lesson is straightforward: do not trust the user's first draft. Help them.

Stop Guessing, Start Searching

Chunk size, overlap percentage, top-k cutoff, and reranker depth interact in ways that are impossible to tune by hand. We spent too many cycles debating whether 256 tokens was better than 512 while ignoring the overlap setting that was actually destroying coherence.

We replaced intuition with Bayesian optimization. Instead of grid search, which wastes compute on obviously bad regions, Bayesian methods build a probabilistic model of what works and actively seek the Pareto frontier that maximizes recall while minimizing latency. For our stack, that meant finding the specific combination of chunk size, overlap, and top-k that gave us ninety-five percent recall without blowing our latency budget. Different use cases landed on different points along that frontier. Customer-facing chatbots prioritized speed. Internal legal research prioritized recall. Automated optimization let us serve both without manual copy-pasting of config files.

The Results

Os números falam por si só. Nosso Recall@10 subiu de setenta e oito por cento para noventa e cinco por cento. A latência p95 caiu de 850 milissegundos para 320 milissegundos. E como o modelo estava finalmente recebendo contexto relevante em vez de ruído, a taxa de alucinação caiu de doze por cento para três por cento. Uma recuperação melhor não torna apenas as respostas mais rápidas. Ela as torna verdadeiras.

O que fazer a seguir

Se você está reconstruindo sua camada de recuperação, comece por aqui:

  • Divida por estrutura do documento, não por contagem de tokens. Adapte sua estratégia de divisão ao formato dos seus dados.
  • Use recuperação híbrida. Combine busca vetorial e BM25, mescle com Reciprocal Rank Fusion e faça o rerank antes de gerar.
  • Expanda as consultas para uma melhor cobertura. Reformule e decomponha antes mesmo da busca ser executada.
  • Construa um golden dataset para testes. Você não pode otimizar o que não mede.
  • Otimize parâmetros com ferramentas automatizadas. A busca Bayesiana encontrará configurações melhores do que o seu instinto.

A recuperação não é um arquivo de configuração que você define uma vez e esquece. É infraestrutura, e infraestrutura merece o mesmo rigor que o código de produção: testes, medições e otimização contínua. Trate-a dessa forma, e seu sistema RAG deixará de ser uma demonstração para se tornar um produto.

Comunidade de aprendizado opcional: GyaanSetu AI