๐ง๐ต๐ฒ ๐๐ฎ๐ถ๐น๐๐ฟ๐ฒ ๐ข๐ณ ๐ ๐ ๐ ๐ฒ๐บ๐ผ๐ฟ๐ ๐ฆ๐ฐ๐ผ๐ฟ๐ถ๐ป๐ด ๐๐ผ๐ฟ๐บ๐๐น๐ฎ
AI agents often pick the wrong memory. They pick memories based on relevance. Relevance is not authority.
An agent asks about VPN access. The system finds a note mentioning VPN. The note is relevant. The official policy is authoritative. The agent picks the note and fails.
I built a formula to fix this. The formula used weights. It looked at authority, scope, and specificity.
The results were a surprise.
- Clean data: It worked perfectly.
- Stress test: It matched old strategies.
- Held-out test: Simple search beat my formula.
The formula is brittle. It needs perfect metadata. If a memory lacks a scope field, the formula fails. A wrong memory with good tags wins. The correct memory with no tags loses.
Ranking rules do not fix bad data. You need a write-time gate. Ensure metadata is valid before storage.
This formula is a diagnostic tool. It exposes where your architecture is weak. It shows the cost of bad metadata.
Source: https://dev.to/zep1997/i-tried-to-turn-agent-memory-authority-into-a-scoring-formula-the-held-out-test-changed-the-claim-4aam Optional learning community: https://t.me/GyaanSetuAi