Most RAG tutorials end at the demo. You chunk by token count, stuff everything into a vector database, and call it a day. That works when a user asks "What is the return policy?" into a clean FAQ. It falls apart when someone pastes half a contract and asks about clause three, or when a developer types an obscure error code into your documentation search.

Fixed token windows cut legal agreements off mid-sentence. Large chunks bury API references under paragraphs of noise. Worst of all, slow retrieval makes users abandon the query before the model even starts generating. We learned this the hard way. When we moved our retrieval layer from hope to measurement, we cut latency by forty percent and pushed recall to ninety-five percent. Here is exactly what changed.

Smart Chunking

Stop thinking about chunk size as a magic number. A 512-token window makes no sense for a legal document where a single clause spans multiple paragraphs, and it is equally useless for API docs where a function signature and its two-line description should stay together. We switched to structure-aware splitting.

For legal text, recursive chunking respects document hierarchy. Clauses stay intact. For API documentation, we use function-aware splitting that keeps signatures, parameters, and examples together as atomic units. Support tickets and conversational data need semantic boundaries, splitting where the topic shifts rather than at an arbitrary character count. The result is that each chunk carries enough context to be useful but not so much that it dilutes the signal. Your embedding model has a limited attention budget. Spend it wisely.

Hybrid Retrieval

Vector search excels at finding conceptually similar content. Ask about slow database queries and it surfaces performance tuning guides. But ask for "Error 0x80070057" and semantic search drifts into unrelated territory because dense embeddings do not handle exact matches well. BM25, on the other hand, nails exact strings and rare terms, yet it has no idea that "latency" and "slow response time" mean the same thing.

We run both in parallel and merge them with Reciprocal Rank Fusion. RRF is simple and effective. It takes the ranked lists from each method and scores documents based on their position, giving strong candidates from either system a fair shot. After fusion, we run a cross-encoder reranker over the combined results and return only the top five. The reranker adds about fifty milliseconds of latency but improved our recall by fifteen percent. That trade-off pays for itself many times over in generation quality.

Query Expansion

Users are terrible at querying. They abbreviate, misspell, or pack three questions into one long ramble. If you search exactly what they typed, you miss the documents they actually need.

We now transform every query before it hits the index. First, we generate multiple rephrased versions of the original question to cover synonyms and alternate phrasings. Second, we decompose complex questions into smaller sub-questions. A query like "Why is the refund failing for international customers and how do I fix it?" becomes two distinct searches: one about international refund failures and one about remediation steps. Query expansion alone moved our recall from seventy-eight percent to ninety-four percent. The lesson is straightforward: do not trust the user's first draft. Help them.

Stop Guessing, Start Searching

Chunk size, overlap percentage, top-k cutoff, and reranker depth interact in ways that are impossible to tune by hand. We spent too many cycles debating whether 256 tokens was better than 512 while ignoring the overlap setting that was actually destroying coherence.

We replaced intuition with Bayesian optimization. Instead of grid search, which wastes compute on obviously bad regions, Bayesian methods build a probabilistic model of what works and actively seek the Pareto frontier that maximizes recall while minimizing latency. For our stack, that meant finding the specific combination of chunk size, overlap, and top-k that gave us ninety-five percent recall without blowing our latency budget. Different use cases landed on different points along that frontier. Customer-facing chatbots prioritized speed. Internal legal research prioritized recall. Automated optimization let us serve both without manual copy-pasting of config files.

The Results

Цифры говорят сами за себя. Наш Recall@10 вырос с 78% до 95%. Задержка p95 снизилась с 850 мс до 320 мс. А поскольку модель наконец-то начала получать релевантный контекст вместо шума, уровень галлюцинаций упал с 12% до 3%. Улучшенный поиск не просто делает ответы быстрее. Он делает их достоверными.

Что делать дальше

Если вы перестраиваете свой слой поиска (retrieval layer), начните отсюда:

  • Разбивайте на чанки по структуре документа, а не по количеству токенов. Подбирайте стратегию разбиения под структуру ваших данных.
  • Используйте гибридный поиск. Сочетайте векторный поиск и BM25, объединяйте их с помощью Reciprocal Rank Fusion и выполняйте переранжирование (rerank) перед генерацией.
  • Расширяйте запросы для лучшего охвата. Перефразируйте и декомпозируйте их еще до запуска поиска.
  • Создайте «золотой» датасет для тестирования. Нельзя оптимизировать то, что вы не измеряете.
  • Оптимизируйте параметры с помощью автоматизированных инструментов. Байесовский поиск найдет более подходящие настройки, чем ваша интуиция.

Поиск (retrieval) — это не конфигурационный файл, который вы один раз настроили и забыли. Это инфраструктура, а инфраструктура требует такого же строгого подхода, как и продакшн-код: тестов, измерений и непрерывной оптимизации. Относитесь к нему именно так, и ваша RAG-система перестанет быть демо-версией и станет полноценным продуктом.

Дополнительное обучающее сообщество: GyaanSetu AI