A recent benchmark shows that a two-layer memory design—RAM-based scratchpad plus a local SQLite-vec vault—delivers median query times under 100 ms while eliminating the $135-per-month bill tied to a popular cloud vector store. Developers building autonomous AI agents can run a purely on-premise setup that, in the benchmark, was faster and incurred no monthly cost.
Why existing memory approaches fall short
AI agents often need to recall thousands of past interactions, facts or tool-call results within a tight response window. Most teams push every embedding into a managed vector database and rely on remote vector search for every lookup. That model scales, but it forces every query through the network, inflating round-trip time and monthly spend. In the benchmark, the cloud service’s median latency sat at 127 ms and the bill topped $135 for the month; the test period also recorded an outage.
Splitting memory into two tiers
The dual-tier architecture separates “working memory” from “long-term storage”:
L1 Scratchpad (RAM)
- Lives entirely in process memory.
- Holds the current task context and the most recent tool calls.
- Stores raw strings; no embeddings are generated.
- Returns results in under 3 ms, comparable to a CPU cache.
L2 Vault (SQLite-vec)
- Persists all other embeddings in a local SQLite database extended with vector search capabilities.
- Handles the full set of 14,726 memories used in the benchmark.
- Returns matches in about 94 ms, comfortably below the 100 ms target for many real-time agents.
- Costs nothing beyond the host machine’s storage.
SQLite-vec is an open-source extension that adds approximate nearest-neighbor search to a standard relational file. Because the database lives on the same machine as the agent, there is no network hop, and the engine reuses existing SQLite indexing tricks to keep lookups fast.
Numbers that matter
| System | Median latency | Monthly cost | Reported reliability |
|---|---|---|---|
| Cloud vector store (Pinecone) | 127 ms | ~$135 | Outages observed |
| Local SQLite-vec vault | 94 ms | $0 | 100 % uptime |
The cost difference is $0 versus approximately $135 per month.
Keeping the vault tidy
A raw dump of all embeddings can dilute relevance. The benchmark’s author introduced a decay system that scores memories by recency and frequency:
- New or frequently accessed items receive a higher weight.
- Items that haven’t been touched in a while gradually lose weight.
- The weighted decay cuts “retrieval noise”—irrelevant matches—by 34 %.
By pruning low-score entries or demoting them in the index, the agent avoids stale data while keeping the vault lean enough for consistent performance.
Takeaway
For AI agents that must juggle thousands of memories under a tight deadline, a RAM-first scratchpad paired with a local SQLite-vec vault offers a pragmatic alternative to fully cloud-based vector stores. The approach trims response times, eliminates recurring cloud fees and delivers uninterrupted service, all while preserving the ability to prune irrelevant data. Adopting the two-tier model can therefore make agents faster, cheaper and more dependable.