After four months of pulling a Retrieval-Augmented Generation (RAG) pipeline out of a Jupyter notebook and into a live service, the author pinpointed five concrete choices that turned a show-piece demo into a system users can actually rely on. The difference shows up in numbers: a simple change to how text is split lifted the retrieval hit rate from 61 % to 83 %, and a modest evaluation set of 200 real queries now catches most regressions before they reach customers.

Why it matters

RAG demos look impressive – they fetch a passage and produce a plausible answer in seconds. In production, the same approach often returns outdated facts, missed error codes, or broken sentences, eroding user trust. The bottleneck is rarely the language model; it is the way content is ingested, indexed, and served. Getting the pipeline right can mean the difference between a product that adds value and one that becomes a liability.

1. Stop using fixed-size chunks

Many prototypes chop every document into 512-token blocks. That works for short texts but shreds technical manuals, support threads, and code snippets. Sentences are split, headings disappear, and the retrieval engine can’t match the context the user expects.

Switch to structure-aware chunking—split at headings, conversation boundaries, or code fences—to preserve semantic units. In the author’s system, this alone raised the fraction of queries that found a relevant passage from 61 % to 83 %. The improvement comes from a data-format change; the underlying model stays the same.

Pure vector search (embedding-based similarity) excels at finding passages with the same meaning, but it stumbles on exact identifiers like error codes, version numbers, or proprietary terminology. A user looking for an error code such as “ERR-XXXX” may get a semantically similar paragraph that doesn’t contain the code at all.

Hybrid search combines a dense vector index with a traditional BM25 index (term-frequency based). By weighting the two scores, the system retrieves items that are both semantically close and contain the exact terms the user typed. For production, hybrid search is a baseline requirement, not a nice-to-have add-on.

3. Handle stale data

Out-of-date pricing tables, policy documents, or firmware release notes quickly destroy credibility. Three practical steps keep the index fresh:

  • Tag every document with a version stamp or timestamp.
  • Apply a recency boost during scoring so newer items outrank older copies.
  • Run nightly incremental re-indexing to pull in changes from source systems.

These safeguards prevent the system from serving a price that was valid last quarter or a policy that has already been superseded.

4. Rerank instead of upgrading models

Upgrading an embedding model only gives a small quality boost, while adding a cross-encoder reranker delivers a much larger jump for less cost.

The production flow fetches 20 cheap candidates using hybrid search, then passes them through the reranker to select the top five. This two-stage approach yields a larger quality jump for a fraction of the expense of a full model upgrade.

5. Build a real evaluation set

You can’t improve what you don’t measure. The author assembled a test suite of 200 genuine user queries, each paired with an expert-crafted answer. Every code change runs against this suite; any regression is caught before deployment.

When a user reports a bad answer, add the query to the evaluation set immediately, turning real-world failures into future safeguards. Continuous logging of every generated answer feeds the evaluation loop, keeping the system aligned with actual usage.

The production pipeline in practice

  • Ingest: Structure-aware chunking preserves headings, code blocks, and conversational turns.
  • Index: Store both dense embeddings and BM25 term statistics.
  • Retrieve: Hybrid search returns 20 candidates, balancing semantic similarity and exact term matches.
  • Rerank: A cross-encoder narrows the list to the five most promising passages.
  • Generate: The LLM receives these top chunks plus their metadata to craft the final answer.
  • Evaluate: Every response is logged; failures are fed back into the 200-query test set.

Stakes and trade-offs

Um pipeline bem ajustado reduz alucinações, melhora a relevância das respostas e diminui o custo de modelos superdimensionados. O benefício é uma maior satisfação do usuário e uma menor carga de suporte. Ignorar essas etapas resulta em um serviço frágil que corrói a confiança na marca e obriga a um caro "apagar incêndios".

O que observar a seguir

À medida que os embeddings de código aberto e os bancos de dados vetoriais amadurecem, a linha entre a recuperação “densa” e “esparsa” se tornará tênue, mas o princípio de combinar correspondência semântica e exata permanece.

Conclusão: Em um sistema RAG, o modelo de linguagem raramente é o gargalo. O verdadeiro trabalho reside em como você segmenta, indexa e apresenta o conteúdo subjacente. Acertar nessas decisões transforma uma demonstração chamativa em um produto confiável.