Most teams build their first retrieval system the same way: slice every document into fixed 512 token chunks, push them

Most teams build their first retrieval system the same way: slice every document into fixed 512-token chunks, push them into a vector database, and hope the embedding model does the hard work. That hope gets you through a demo. It does not survive contact with real users.

In production, a legal contract falls apart when you sever a liability clause from its exceptions. API documentation turns useless when a code sample gets detached from its function signature. A customer support thread becomes noise when you rip a single complaint out of its conversational history. The problem is rarely the language model sitting at the end of the pipeline. The problem is what you feed it.

We learned this the hard way. Our initial retrieval layer looked standard but behaved inconsistently. So we rebuilt it around a simple idea: treat retrieval as measured infrastructure, not magic. Here is exactly what changed, and how we pushed recall to 95 percent while cutting the 95th-percentile latency from 850 ms to 320 ms.

The Fixed-Chunk Trap

Uniform token counts are easy to code and easy to explain. That convenience masks a basic truth: documents have structure. When you ignore that structure, you destroy signal.

Consider a ten-page master service agreement. A fixed 512-token slice will land mid-obligation, splitting a clause from the very cap table that limits it. The retrieval step then returns half a thought. The generator hallucinates the rest. In API documentation, a chunk that is too large dilutes the embedding with boilerplate headers, burying the specific method a developer needs. In support tickets, a fixed window treats a conversation as a bag of sentences, stripping away the back-and-forth that reveals what actually failed.

We stopped treating chunk size as a hyperparameter we guessed. We started treating it as a mapping exercise between the document type and the information architecture inside it.

Match Your Chunking to the Data

The fix is not one perfect chunk size. The fix is three distinct strategies tuned to three distinct data shapes.

Legal documents now go through recursive splitting. The algorithm first looks for the largest natural boundaries—sections, then subsections, then numbered clauses—and only falls back to smaller splits when necessary. This keeps a termination clause attached to its survival conditions. The retrieval step sees complete logical units, which sharply reduces the model’s temptation to invent missing exceptions.

API and code documentation get structure-aware chunking. Markdown headers, code fences, and parameter tables are parsed as atomic units. We do not split inside a code block. We keep docstrings adjacent to their signatures. The result is that a query for a specific class method retrieves the full context a developer needs: the description, the typed parameters, and the working example.

Support and conversational data use semantic chunking. Instead of counting tokens, we look for shifts in topic or intent. If a customer describes a bug in message three and pastes a stack trace in message seven, we chunk by meaning, not by message index. The retrieval layer then returns the full arc of the problem rather than a orphaned sentence.

Why Vector Search Alone Fails

Even perfect chunks die in a pure vector search. Dense embeddings excel at capturing meaning and synonymy, but they are notoriously fuzzy on exact strings. If an engineer searches for the precise error code ERR_CONNECTION_REFUSED, vector similarity might return a dozen conceptual neighbors and miss the exact match buried at rank fourteen.

Keyword search with BM25 has the opposite problem. It finds exact tokens but misses semantic intent. A user asking “why is my database down” will never match a document that says “troubleshooting connection timeouts.”

We now run both. Vector and keyword results are fed into Reciprocal Rank Fusion, which blends the two ranked lists without requiring calibrated scores. The fused list is then passed through a cross-encoder reranker. The reranker is slower than the initial retrieval, but it is far more precise because it judges query-document relevance directly rather than through compressed embeddings. That hybrid pipeline alone lifted our recall by 15 percent.

Fixing Bad Queries Before They Hit the Index

用户不会编写理想的搜索查询。他们会粘贴截断的日志行。他们会输入“坏了”。他们会使用你的文档从未采用过的术语。如果你信任原始查询，你就是在信任噪声。

我们现在在将每个传入查询发送到检索层之前，将其扩展为三到五个变体。其中一个变体可能是直接的改写。另一个可能是假设的理想文档标题。第三个则会剥离对话式填充词并隔离技术关键词。每个变体都会进行嵌入（embedding）并进行搜索。然后我们对候选池进行去重和合并。

这并非没有代价。这些额外的嵌入调用会产生费用并增加几毫秒的延迟。但对召回率（recall）的影响是巨大的：通过在检索前扩展查询，我们将召回率从 78% 提升到了 96%。因为更好的检索缩减了生成窗口并让模型立足于正确的上下文，我们最终节省了下游的成本。一个略微昂贵的检索步骤比一个漫长且产生幻觉的生成步骤更便宜。

停止猜测，开始搜索。

一旦我们确定了正确的切片（chunking）、混合检索和查询扩展，我们仍然面临着组合爆炸的问题。切片大小、切片重叠度、top-k 检索深度、重排序（reranker）截断值以及融合权重都会相互影响。手动网格搜索将耗费数周时间，且仍可能让我们陷入局部最优解。

我们转向使用贝叶斯优化（Bayesian optimization）来探索这个空间。搜索算法不再是穷举测试每种组合，而是对哪些配置可能表现良好保持一种“信念”，并逐步缩小到有希望的区域。

输出结果并不是单一的完美设置，而是一个帕累托前沿（Pareto frontier）的选择。在这一端，我们有一个针对高吞吐量 API 支持端点优化的精简配置：快速推理、中等召回率和尽可能低的延迟。在另一端，我们有一个针对法律审查的激进配置：更深层次的检索、更重的重排序以及更紧密的重叠，通过牺牲毫秒级延迟来换取彻底性。由于前沿是明确的，我们可以为产品选择正确的点，而不是假装“一刀切”。

数据究竟如何

这些变化使系统从一个脆弱的原型转变为一个经过衡量的生产流水线。

Recall at ten（前十项召回率）从 78% 提高到了 95%。这意味着当正确答案存在于我们的语料库中时，我们每 20 次中能找到 19 次。

P95 延迟（95th percentile latency）从 850 毫秒降至 320 毫秒。混合栈在理论上听起来更重，但更智能的索引、更小的重排序器以及仅在需要时提供激进切片的能力，使整个系统变得更快。

幻觉率（由人工标注员在预留的金标准数据集上进行跟踪）从 12% 下降到 3%。当模型接收到完整且相关的上下文时，它就不再捏造事实。

单次查询成本从 $0.008 降至 $0.005。更好的检索意味着更短、更集中的 LLM 提示词（prompts）和更少的恢复尝试。查询扩展带来的额外嵌入支出，在生成环节节省的成本面前显得微不足道。

构建金标准数据集，并将检索视为代码

如果你从中能学到一点，那应该是“测量”的纪律性。我们构建了一个包含真实问题和验证过的答案位置的小型金标准数据集（golden dataset）。在任何更改进入生产环境之前，它都会在该数据集上运行。召回率和延迟是实时监控的，而不是在笔记本中凭感觉观察。

检索不是研究演示，它是基础设施。它值得拥有单元测试、回归基准测试和自动化优化，就像你的技术栈中的其他部分一样。根据文档结构进行切片，而不是仅凭对 token 数量的迷信。将向量搜索和关键词搜索结合起来，并使用重排序器。扩展用户实际编写的查询。然后，让搜索算法来调节参数，而不是依靠你的直觉。

我们描述的流水线并非理论。你可以在这里阅读原始文章。如果你想与关注这些内容的社区讨论检索工程，GyaanSetu AI group 随时欢迎你。