Most RAG tutorials end at the demo. You chunk by token count, stuff everything into a vector database, and call it a day. That works when a user asks "What is the return policy?" into a clean FAQ. It falls apart when someone pastes half a contract and asks about clause three, or when a developer types an obscure error code into your documentation search.

Fixed token windows cut legal agreements off mid-sentence. Large chunks bury API references under paragraphs of noise. Worst of all, slow retrieval makes users abandon the query before the model even starts generating. We learned this the hard way. When we moved our retrieval layer from hope to measurement, we cut latency by forty percent and pushed recall to ninety-five percent. Here is exactly what changed.

Smart Chunking

Stop thinking about chunk size as a magic number. A 512-token window makes no sense for a legal document where a single clause spans multiple paragraphs, and it is equally useless for API docs where a function signature and its two-line description should stay together. We switched to structure-aware splitting.

For legal text, recursive chunking respects document hierarchy. Clauses stay intact. For API documentation, we use function-aware splitting that keeps signatures, parameters, and examples together as atomic units. Support tickets and conversational data need semantic boundaries, splitting where the topic shifts rather than at an arbitrary character count. The result is that each chunk carries enough context to be useful but not so much that it dilutes the signal. Your embedding model has a limited attention budget. Spend it wisely.

Hybrid Retrieval

Vector search excels at finding conceptually similar content. Ask about slow database queries and it surfaces performance tuning guides. But ask for "Error 0x80070057" and semantic search drifts into unrelated territory because dense embeddings do not handle exact matches well. BM25, on the other hand, nails exact strings and rare terms, yet it has no idea that "latency" and "slow response time" mean the same thing.

We run both in parallel and merge them with Reciprocal Rank Fusion. RRF is simple and effective. It takes the ranked lists from each method and scores documents based on their position, giving strong candidates from either system a fair shot. After fusion, we run a cross-encoder reranker over the combined results and return only the top five. The reranker adds about fifty milliseconds of latency but improved our recall by fifteen percent. That trade-off pays for itself many times over in generation quality.

Query Expansion

Users are terrible at querying. They abbreviate, misspell, or pack three questions into one long ramble. If you search exactly what they typed, you miss the documents they actually need.

We now transform every query before it hits the index. First, we generate multiple rephrased versions of the original question to cover synonyms and alternate phrasings. Second, we decompose complex questions into smaller sub-questions. A query like "Why is the refund failing for international customers and how do I fix it?" becomes two distinct searches: one about international refund failures and one about remediation steps. Query expansion alone moved our recall from seventy-eight percent to ninety-four percent. The lesson is straightforward: do not trust the user's first draft. Help them.

Stop Guessing, Start Searching

Chunk size, overlap percentage, top-k cutoff, and reranker depth interact in ways that are impossible to tune by hand. We spent too many cycles debating whether 256 tokens was better than 512 while ignoring the overlap setting that was actually destroying coherence.

We replaced intuition with Bayesian optimization. Instead of grid search, which wastes compute on obviously bad regions, Bayesian methods build a probabilistic model of what works and actively seek the Pareto frontier that maximizes recall while minimizing latency. For our stack, that meant finding the specific combination of chunk size, overlap, and top-k that gave us ninety-five percent recall without blowing our latency budget. Different use cases landed on different points along that frontier. Customer-facing chatbots prioritized speed. Internal legal research prioritized recall. Automated optimization let us serve both without manual copy-pasting of config files.

The Results

숫자는 명확하게 말해줍니다. Recall@10은 78%에서 95%로 상승했습니다. p95 지연 시간(latency)은 850밀리초에서 320밀리초로 감소했습니다. 그리고 모델이 노이즈 대신 마침내 관련성 있는 컨텍스트를 받게 되면서, 환각(hallucination) 발생률은 12%에서 3%로 떨어졌습니다. 더 나은 검색은 단순히 답변을 빠르게 만드는 것에 그치지 않습니다. 답변을 정확하게 만듭니다.

향후 조치 사항

검색 레이어를 재구축하고 있다면, 여기서부터 시작하세요:

  • 토큰 수가 아닌 문서 구조에 따라 청킹하세요. 데이터의 형태에 맞춰 분할 전략을 설정해야 합니다.
  • 하이브리드 검색을 사용하세요. 벡터 검색과 BM25를 결합하고, Reciprocal Rank Fusion으로 병합한 뒤, 답변을 생성하기 전에 재순위화(rerank)를 수행하세요.
  • 더 넓은 범위를 커버할 수 있도록 쿼리를 확장하세요. 검색을 실행하기 전에 쿼리를 재구성하거나 분해하세요.
  • 테스트를 위한 골든 데이터셋(golden dataset)을 구축하세요. 측정할 수 없는 것은 최적화할 수 없습니다.
  • 자동화 도구로 파라미터를 최적화하세요. 베이지안 탐색(Bayesian search)은 직관보다 더 나은 설정을 찾아줄 것입니다.

검색은 한 번 설정하고 잊어버리는 설정 파일이 아닙니다. 검색은 인프라이며, 인프라는 테스트, 측정, 지속적인 최적화와 같이 프로덕션 코드와 동일한 엄격함이 필요합니다. 그렇게 다룬다면, 여러분의 RAG 시스템은 단순한 데모를 넘어 하나의 제품이 될 것입니다.

선택 사항 학습 커뮤니티: GyaanSetu AI