A developer’s RAG prototype refused to answer any query with a cosine similarity below 0.50. It worked flawlessly—until the embedding model was swapped. The guard then let wrong answers slip through silently. The incident proves that a hard-coded similarity threshold can collapse across models, a risk that threatens any system that relies on embedding similarity for safety checks.
Why the threshold mattered
Retrieval-augmented generation (RAG) pipelines often use a similarity guard: if the cosine similarity between a query and its nearest document falls under a preset number, the system aborts the answer. The guard keeps the model from hallucinating when the retrieved context is weak. In the original setup, a 0.50 threshold kept the system honest—unanswerable queries scored below the line, answerable ones above it.
When the embedding backend changed, the same 0.50 cut-off no longer separated the two groups. The pipeline began returning confident, but incorrect, responses without any crash or explicit error. The failure escaped standard ranking metrics and showed up only when a human noticed the drift.
Geometry isn’t universal
Every embedding model maps language into a high-dimensional space with its own geometry. Cosine similarity values therefore mean different things from one model to the next. A score of 0.50 can sit at the edge of a clear gap for one model and deep inside the overlap for another.
- Voyage-3 – the 0.50 line sits between low-scoring unanswerable queries and high-scoring answerable ones. The guard works as intended.
- BGE-Small – many unanswerable queries score above 0.50, so the guard never fires. Raising the cut-off to 0.70 restores the safety margin.
- Hashing-64 – scores for answerable and unanswerable queries intermix so tightly that no single threshold separates them; the model is too weak to support a similarity guard at all.
These cases illustrate a broader truth: a threshold belongs to a specific model-data pair, not a universal rule.
The hidden cost of a constant
Similarity thresholds appear in many downstream tasks:
- semantic caching
- duplicate detection
- document relevance checks
- entity matching
Using a constant value assumes that all embedding models share the same score distribution—a dangerous assumption. When the assumption fails, systems silently produce false-confident output, eroding user trust and feeding downstream decisions with bogus data.
Calibrating per-model guards
The remedy is straightforward: never ship a hard-coded constant. Treat the threshold as a hyper-parameter that must be tuned for every new embedding model.
- Assemble a modest, labeled validation set covering both answerable and unanswerable queries.
- Compute cosine similarities for each query-document pair using the target model.
- Plot the two distributions or calculate the false-confident rate—the proportion of unanswerable queries that exceed the candidate threshold.
- Choose the smallest similarity value that keeps the false-confident rate below an acceptable risk level.
Because the goal is to prevent over-confidence, traditional ranking metrics like mean reciprocal rank (MRR) are insufficient. The false-confident rate directly measures the guard’s failure mode.
Counter-point: “Some models work out of the box”
It is true that certain well-behaved models, such as Voyage-3 in the example, happen to line up with the 0.50 guard. That does not guarantee future stability. Model updates, fine-tuning, or even shifts in the underlying corpus can alter the similarity distribution, breaking the guard again. Relying on a single lucky alignment invites complacency.
What to watch next
-
-
-
Takeaway
A similarity threshold is not a universal safety switch; it is a model-specific guard that must be calibrated each time you change the embedding backend or the data it sees. Ignoring this fact lets RAG systems slip into silent hallucination, undermining the very purpose of the guard. The only reliable path forward is systematic per-model calibration and ongoing monitoring of the false-confident rate.