The VRAM Cliff — Finding the Real Context Ceiling for a 27B Model on 16GB

3 September 2026 · local-llm

If you run a large quantised model on a consumer GPU, you will eventually meet this failure: everything works, you raise the context size a little, and throughput collapses by a factor of ten. No error. No out-of-memory. No warning in the log. Just a model that went from usable to unusable because of one number in a config file.

This is a writeup of finding that cliff on a 16GB card, including the part where the first explanation turned out to be wrong.

The setup

• GPU: 16GB consumer card
• CPU: AMD Ryzen 5 9600X, 6 cores / 12 threads
• Runtime: llama.cpp server
• Model: a 27B dense model at IQ4_XS, roughly 12.8GB on disk — a mixed-precision quant with attention tensors at IQ4_XS and FFN tensors at IQ3_S

The working configuration:

-t 6 --fit off -ngl 999 -np 1 -fa on --load-mode none
-b 2048 -ub 512 -ctk q4_0 -ctv q4_0
-c 76000 -n 16384
--spec-type draft-mtp --spec-draft-n-max 3
--jinja

Measured live at that configuration: 80-99 tok/s generation, ~1180 tok/s prompt processing on a 16,000-token system prompt, with multi-token prediction hitting about 85% draft acceptance.

Then raise -c from 76000 to 80000, change nothing else:

Same model. Same flags. Same card. One number.

The first explanation, which was wrong

An earlier round of testing on a different quant of the same model produced a much more dramatic version of this: prompt processing on a real 6,013-token prompt fell from 1,436.8 tok/s to 72.5 tok/s — a 20x collapse, reproduced twice.

The variable that had changed was batch size. -b and -ub had been cut from 2048 to 512 in an attempt to make a larger quant fit. The obvious conclusion: I-quants are more compute-expensive per token than K-quants, small batches amortise that cost badly, therefore small batch plus I-quant equals collapse.

That conclusion was wrong, and it took a deliberately isolated re-test to show it. Same quant file, full GPU offload, no CPU-offloaded tensors, -c 8192:

A 5% difference. Well inside noise. The batch size was never the culprit.

What had actually been different in the original failing test was that it combined the small batch with --override-tensor CPU-offloaded FFN layers at full context. Small batches mean more, smaller transfers; CPU-offloaded layers mean every one of those transfers crosses PCIe. The two together produced the collapse. Neither alone did.

I am including this because the disproved hypothesis is the useful part. "Small batch is slow with I-quants" is a plausible-sounding rule that would have been carried forward forever and quietly cost throughput in configurations where it does not apply.

The convergence clue

Pushing that same fully-GPU-offloaded configuration to -c 65536 produced a different failure — and this one was informative:

• VRAM headroom dropped from ~1.3GB free at -c 8192 to 150-200MB free
• Prompt processing: 166 tok/s
• Generation: 2.7-3.2 tok/s
• Identical with multi-token prediction on and off, so MTP was not the cause

Then the detail that explains everything. Across every collapsed test — a different model, a different quant, a different number of CPU-offloaded layers — prompt processing converged on the same ~166 tok/s.

A model-specific or quant-specific problem would not do that. Different models have different compute profiles and would degrade to different numbers. Landing on one identical rate regardless of what is running means the bottleneck is not the model at all. It is whatever the data is travelling through once it stops fitting.

The most likely explanation: once free VRAM drops below some threshold, Windows silently pages the overflow into system memory rather than failing the allocation. From that point the ceiling is system-memory bandwidth, which is the same number no matter which model is being paged.

This is a platform-level ceiling, not a model property. That reframing is what makes the cliff findable, because it means the fix is never "try a different quant" — it is "create headroom".

Idle free VRAM does not predict the cliff

The obvious method is to watch nvidia-smi, find how much VRAM is free at idle, and reason about how much context that buys. It does not work. One tested configuration showed more idle free VRAM than a known-good baseline and still collapsed.

The reason is that the determinant is not resident memory. It is the transient compute buffer allocated during prefill, which exists only while a prompt is being processed and is invisible in a reading taken at rest.

The practical consequence: you cannot calculate your ceiling, you have to measure it.

Bisecting the ceiling

The method that works is boring and reliable. Hold every variable fixed except context, and bisect on throughput.

With a fixed 20-layer --override-tensor FFN offload set:

So 75,776 was the real ceiling for that layer set — a free 15.7% more context than the 65,536 it had been running at, with no speed penalty and no config change beyond the number itself. That headroom was simply being left on the table because nobody had measured where the edge actually was.

For the currently deployed configuration the same bisection put the edge between 76,000 and 80,000, which is why -c 76000 is a hard ceiling rather than an arbitrary round number.

More offloaded layers is not monotonically better

If the cliff is about headroom, the fix is to move more work off the GPU. But more is not simply better. Holding context fixed at 131,072 and bisecting on the number of CPU-offloaded FFN layers:

Two things worth noting. The boundary is not clean — 40 layers sits in a partial-degradation zone that is neither baseline nor full collapse, and is a bad place to land. And offloading everything is not optimal: 64 layers buys faster prompt processing but slower generation than 48, because the extra offloaded layers cost generation time without buying headroom that was still needed.

48 was the sweet spot, not 64. If you are past the threshold, offloading more just burns generation speed.

Choosing which layers to offload

--override-tensor takes a regex, so you choose which tensors move. Two rules came out of this:

Offload FFN tensors, not attention. FFN layers do not touch the KV cache, so they tolerate the PCIe round trip far better than attention layers would.

Prefer the biggest K-quant layers. Modern dynamic quants mix K-quant and I-quant tensors per layer by importance. I-quant layers are already more compute-expensive, and are better left on the GPU — which is what the failed I-quant test showed independently.

Picking them is mechanical rather than guesswork:

pip install gguf

from gguf import GGUFReader
r = GGUFReader("model.gguf")
ffn = [t for t in r.tensors if ".ffn_" in t.name]
for t in sorted(ffn, key=lambda t: t.n_bytes, reverse=True)[:40]:
    print(t.name, t.tensor_type, t.n_bytes)

Rank by byte size, filter to K-quant types, take from the top until you have freed the headroom you need, and leave the draft head alone if the model has one. That produces a regex like:

--override-tensor "blk\.(24|25|26|27|38|40|50|51|52|53|54|55|56|57|58|59|60|61|62|63)\.ffn_.*=CPU"

Set the thread count to your actual core count

Once FFN layers are on the CPU, the CPU is genuinely doing work. -t 4 on a 6-core part was leaving performance unused; -t 6 matches the hardware. This only starts mattering after you offload — before that it is nearly irrelevant.

What this costs, honestly

Multi-token prediction was worth keeping throughout. In the hybrid GPU/CPU configuration it took generation from 13.14 tok/s to 21.27 tok/s — a 62% gain at 83.3% draft acceptance. It keeps paying even when layers are split across devices.

The tradeoff that is not free: the current deployment leaves only about 230MB of VRAM free at idle. Local speech-to-text needs roughly 720MB of GPU memory, so it no longer fits and falls back to CPU. That was a deliberate trade for roughly 2x generation speed and a better quant, not an oversight. Every headroom decision on a card this size takes memory away from something.

The method, compressed

1. Do not calculate your ceiling. Measure it.
2. Bisect on one variable at a time, holding everything else fixed.
3. Treat an identical degraded throughput number across different configurations as a signal that you have left model-specific territory.
4. Ignore idle free VRAM. The transient prefill buffer is what binds.
5. Offload FFN, not attention. Prefer large K-quant layers.
6. Offload the minimum that clears the cliff, not the maximum.
7. Write down the disproved hypotheses too. They are the ones that come back.