Local large language models feel lightning-fast at first. You load a 7B or 13B parameter model, fire off a short prompt, and tokens stream across the screen at a comfortable clip. Then you paste a long code block, or your chat history swells across a dozen turns, and the model begins to crawl. The slowdown is rarely gentle. It is a cliff. One moment the GPU is churning out tokens; the next, your system monitor shows memory pressure building and generation slows to a stutter. You cannot predict exactly when this will happen with a tidy formula. Your only reliable guide is the hardware itself.

The Hidden Cost of Context

Every token you generate adds state to the KV cache. This cache stores the keys and values computed during the prefill and generation phases, and it lives in memory alongside your model weights, attention buffers, and runtime overhead. On a typical consumer GPU with 12 GB or 16 GB of VRAM, the KV cache eventually competes for space with everything else. When dedicated video memory fills, the operating system does not throw an error and stop. It quietly spills the overflow into shared memory, shuttling data between the GPU and system RAM across the PCIe bus. That bus is fast for file transfers, but it is glacial compared to the memory bandwidth inside a graphics card. The result is not a minor dip in performance. It is a collapse.

Three Signals That the Cliff Has Arrived

Watch your hardware monitors while the model runs. You will see three clear signals once the performance cliff hits.

  • Shared VRAM climbs. This is memory that the GPU driver has pushed out of dedicated video RAM into the pool managed by the host operating system. The moment this metric rises above zero, you have crossed the line.
  • System RAM usage swells. The overflow has to land somewhere, and that destination is your main memory. If your RAM usage grows while the model generates tokens, data is being offloaded from the GPU.
  • Eval speed drops by half or more. A 10% slowdown might mean thermal throttling or background processes. A 50% drop, or worse, means the bottleneck has shifted from tensor cores to memory bandwidth and PCIe latency. When you see generation fall from double digits to single digits, you have already fallen off the cliff.

Why Your Quick Benchmark Is Probably Lying

A short smoke test will give you false confidence. If you benchmark the model with a hundred-token prompt, see healthy throughput, and call it a day, you have measured the honeymoon phase. The KV cache is nearly empty. The layers have not been stressed by a long prefill. The true footprint only reveals itself after the model has processed a substantial prompt and the cache has filled to its real working size. You must test with deep prefill and long generation runs. Let the context actually accumulate. Only then will the memory pressure stabilize and show you the genuine limit.

Finding Your Limit with llama.cpp

If you are running models through llama.cpp, you can measure your wall with simple arithmetic and a patient test run.

1. Measure shared memory usage.
Record your baseline dedicated VRAM with a minimal prompt, then run a long-context task and note the peak. Subtract baseline from peak. The difference is what has spilled out of your GPU into shared system memory.

2. Calculate your RAM delta.
Do the same subtraction for system RAM. Subtract your baseline RAM from the peak RAM during the long run. This number tells you exactly how much data has been pushed from the video card to your main memory. It quantifies the leak across the bus.

3. Time the eval speed collapse.
Compare your baseline tokens-per-second rate against the rate after the model has chewed through a long document. You might watch a model cruise at seventeen tokens per second when the context is fresh, then deliver only two tokens per second once the cache has bloated. That fifteen-token drop is your canary in the coal mine.

Triangulating the Breaking Point

To map the curve accurately, do not settle for one lonely data point. Run three distinct trials at 16,000 tokens, 32,000 tokens, and 65,000 tokens. Two points might suggest a line, but two dots are just a guess. The third point proves whether you are looking at measurement noise or a real memory wall. Subtract the results between runs to calculate how much extra memory each additional thousand tokens consumes on your specific combination of model, quantization layer, and GPU.

Once you have that slope, you can project forward. Take your per-token cost, multiply it by the target context length, divide by 1024 to move between units, and add the result to your base model VRAM load. The equation looks like this:

Model VRAM load + (tokens × memory per token ÷ 1024) = Theoretical VRAM usage

This projection is not prophecy. It is a guidepost derived from actual behavior. Use it to estimate your ceiling before you commit to a full production run.

Why Paper Formulas Fail, and What Quantization Can Fix

Textbook formulas ignore the messy reality of local inference. Different architectures allocate attention buffers differently. Your operating system reserves VRAM for the display driver, compositor, and CUDA context. Driver versions change how aggressively they use shared memory. A theoretical equation cannot know how much VRAM is actually free on your machine at 2:00 PM with a browser full of tabs open. You have to run the model on your specific hardware and watch the meters.

Quantization offers partial relief. Moving the KV cache from f16 to q8_0 halves its memory footprint while keeping precision high enough for nearly all practical tasks. That change buys you headroom. It does not grant immunity. The cache still grows linearly with every token you feed in. Eventually, even the reduced size overwhelms your available dedicated memory and the spillover to system RAM begins. The pressure only stops when the context window is capped or the data stops moving.

The Real Takeaway

Do not trust marketing slides, parameter counts, or back-of-the-envelope math. Load the model. Open your system monitor. Run a 65,000-token thread, watch the RAM climb, and count the tokens per second. The numbers that appear on your specific screen, on your specific GPU, are the only numbers that matter. Context always wins. Your job is to know exactly when it wins on your machine.