After four months of pulling a Retrieval-Augmented Generation (RAG) pipeline out of a Jupyter notebook and into a live service, the author pinpointed five concrete choices that turned a show-piece demo into a system users can actually rely on. The difference shows up in numbers: a simple change to how text is split lifted the retrieval hit rate from 61 % to 83 %, and a modest evaluation set of 200 real queries now catches most regressions before they reach customers.

Why it matters

RAG demos look impressive – they fetch a passage and produce a plausible answer in seconds. In production, the same approach often returns outdated facts, missed error codes, or broken sentences, eroding user trust. The bottleneck is rarely the language model; it is the way content is ingested, indexed, and served. Getting the pipeline right can mean the difference between a product that adds value and one that becomes a liability.

1. Stop using fixed-size chunks

Many prototypes chop every document into 512-token blocks. That works for short texts but shreds technical manuals, support threads, and code snippets. Sentences are split, headings disappear, and the retrieval engine can’t match the context the user expects.

Switch to structure-aware chunking—split at headings, conversation boundaries, or code fences—to preserve semantic units. In the author’s system, this alone raised the fraction of queries that found a relevant passage from 61 % to 83 %. The improvement comes from a data-format change; the underlying model stays the same.

Pure vector search (embedding-based similarity) excels at finding passages with the same meaning, but it stumbles on exact identifiers like error codes, version numbers, or proprietary terminology. A user looking for an error code such as “ERR-XXXX” may get a semantically similar paragraph that doesn’t contain the code at all.

Hybrid search combines a dense vector index with a traditional BM25 index (term-frequency based). By weighting the two scores, the system retrieves items that are both semantically close and contain the exact terms the user typed. For production, hybrid search is a baseline requirement, not a nice-to-have add-on.

3. Handle stale data

Out-of-date pricing tables, policy documents, or firmware release notes quickly destroy credibility. Three practical steps keep the index fresh:

  • Tag every document with a version stamp or timestamp.
  • Apply a recency boost during scoring so newer items outrank older copies.
  • Run nightly incremental re-indexing to pull in changes from source systems.

These safeguards prevent the system from serving a price that was valid last quarter or a policy that has already been superseded.

4. Rerank instead of upgrading models

Upgrading an embedding model only gives a small quality boost, while adding a cross-encoder reranker delivers a much larger jump for less cost.

The production flow fetches 20 cheap candidates using hybrid search, then passes them through the reranker to select the top five. This two-stage approach yields a larger quality jump for a fraction of the expense of a full model upgrade.

5. Build a real evaluation set

You can’t improve what you don’t measure. The author assembled a test suite of 200 genuine user queries, each paired with an expert-crafted answer. Every code change runs against this suite; any regression is caught before deployment.

When a user reports a bad answer, add the query to the evaluation set immediately, turning real-world failures into future safeguards. Continuous logging of every generated answer feeds the evaluation loop, keeping the system aligned with actual usage.

The production pipeline in practice

  • Ingest: Structure-aware chunking preserves headings, code blocks, and conversational turns.
  • Index: Store both dense embeddings and BM25 term statistics.
  • Retrieve: Hybrid search returns 20 candidates, balancing semantic similarity and exact term matches.
  • Rerank: A cross-encoder narrows the list to the five most promising passages.
  • Generate: The LLM receives these top chunks plus their metadata to craft the final answer.
  • Evaluate: Every response is logged; failures are fed back into the 200-query test set.

Stakes and trade-offs

פייפליין מכויל היטב מפחית הזיות, משפר את רלוונטיות התשובות ומקצץ את העלות של מודלים בעלי משאבים מוגזמים. היתרון הוא שביעות רצון גבוהה יותר של המשתמשים ועומס תמיכה נמוך יותר. התעלמות מהשלבים הללו יוצרת שירות שביר ששוחק את האמון במותג ומאלץ "כיבוי שריפות" יקר.

מה כדאי לעקוב אחריו בהמשך

ככל שה-embeddings בקוד פתוח ובסיסי הנתונים הוקטוריים מבשילים, הגבול בין שליפה "צפופה" (dense) ל"דלילה" (sparse) יטשטש, אך העיקרון של שילוב התאמה סמנטית והתאמה מדויקת נשאר בעינו.

שורה תחתונה: במערכת RAG, מודל השפה הוא לעיתים רחוקות צוואר הבקבוק. העבודה האמיתית טמונה באופן שבו אתם מחלקים, מאנדקסים ומציגים את התוכן הבסיסי. קבלת החלטות נכונות בתחומים אלו הופכת דמו מרשים למוצר אמין.