Article: Structuring memory by type cuts retrieved tokens by about 40%.
Why a flat memory store breaks down
Most beginner tutorials teach an LLM agent to “remember” by appending every new piece of information to a single list and feeding that list back to the model each turn. The code is literally three lines long, and it produces a working demo. In practice, the list grows unchecked. Two symptoms appear:
- The agent treats outdated data as still true, for example delivering an ETA that expired hours ago.
- The context window fills with trivia that never influences the answer, inflating API costs and slowing response times.
A plain vector store or simple key-value cache can’t tell a user’s job title apart from a temporary project status. When the agent runs a semantic search, the similarity algorithm may surface an old ETA simply because the query contains the same words, even though the datum is no longer relevant.
Structured memory: four buckets, one purpose
The remedy is to stop treating memory as a monolith and start classifying each entry into one of four categories:
- User facts – stable attributes such as a user’s role, preferred language, or security clearance. These rarely change and can be cached for the entire session.
- Feedback – explicit rules the agent must obey, e.g., “never reveal database passwords” or “avoid humor in compliance queries.” Because they govern behavior, they belong in the system prompt rather than the searchable pool.
- Project state – fast-moving data like current ETAs, task progress, or temporary tokens. This bucket needs an expiry check; once a timestamp falls outside a defined window the entry should be purged.
- References – pointers to external services, document IDs, or API endpoints. They are not content to be displayed but routes to fetch fresh data when needed.
Mem0 lets developers attach arbitrary metadata to each memory record. By indexing on the “kind” field, a query can first filter for the relevant bucket before the LLM decides how to use the result.
Two-step retrieval with Mem0
- Pull memories by kind – A short filter query asks Mem0 for “all feedback” or “project-state entries newer than a short interval.” The result set is already trimmed to the appropriate category.
- Let the LLM decide – The filtered snippets are inserted into the prompt along with the user’s current question. The model can now reason about them without sifting through irrelevant facts.
A concrete illustration: instead of waiting for a semantic match to surface the rule “do not mock the database,” the developer injects that rule directly into the system prompt at session start and caches it for the whole interaction. Even if the user’s query contains no explicit reference to databases, the model already knows the constraint.
Practical tricks that keep costs down
- Cache feedback rules – Store the rule set once per session and reuse it rather than re-searching on every turn. This reduces token usage each round.
- Skip project-state searches when irrelevant – If the user asks a purely conceptual question (“What is the difference between supervised and reinforcement learning?”), there’s no need to pull any ETA or task progress data.
By applying these two habits, token usage can be reduced by about 40% compared with a naïve flat memory approach. The savings translate directly into lower API bills and faster turnaround, especially for agents that stay alive for many exchanges.
Who gains, who worries
Winners – Teams building customer-support bots, internal workflow assistants, or any multi-turn LLM interface. They get more reliable answers, avoid embarrassing mistakes caused by outdated data, and stretch their budgets further.
Takeaway
If you want an LLM agent that stays sharp over long sessions, stop stuffing every fact into a single context window. Tag each memory as user fact, feedback, project state, or reference, enforce expiries where needed, and let a tool like Mem0 do the heavy lifting. The result is fresher answers, fewer stray tokens, and a noticeable cut in operating costs.