Pick the Quant on Quality, Then Tune It for Speed

10 September 2026 · local-llm

Speed is the easiest thing to measure about a local model, so it is usually the first thing measured, and often the only thing. llama-bench will hand you a prompt-processing and a generation number in a couple of minutes. Quality takes an afternoon of actually using the thing.

That asymmetry quietly decides which model you run. Over a run of quant swaps on one 16GB card, the two-pass order below produced better outcomes than ranking candidates by throughput, and it is worth writing down because the first pass is the one everyone skips.

Pass one: pick the quant on quality. Compare candidate files against each other on the work you actually do, and choose a winner.

Pass two: tune that winner for speed and context. Flags, batch sizes, context length, offload — all of it, on the one file that already won.

The rule that makes it work is that pass two never reopens pass one. Once a quant has won on quality, a faster alternative is not a reason to go back and re-litigate; it is a reason to make the winner faster.

Speed-first would have picked the wrong file

Two community quants of the same 27B model, both nominally IQ4_XS, both built to fit a 16GB card. On throughput they were close enough to call a tie:

Ranked on speed alone, A arguably wins: comparable throughput and 148MB more headroom, which on a card this tight is the difference between local speech-to-text fitting on the GPU or falling back to the CPU.

B was adopted anyway, because its published divergence from the full-precision model was materially lower — a KLD of 0.035 against 0.055. That is a quality difference the throughput columns cannot see, on two files that a speed-ranked table would have called equivalent.

The headroom cost was real and was accepted deliberately: speech-to-text now runs on CPU. That is the shape of the trade this method keeps making — pay a measurable cost in pass two to keep a quality win from pass one.

Benchmark the workload, not the benchmark

The comparison above was not run on llama-bench. It was run by giving both candidates the same real agentic tasks — write a script and run it, diagnose a failing test suite and fix it — and watching which one completed them.

Synthetic numbers are still useful, but they answer a narrower question than they appear to. The clearest evidence of that came from thread count.

An early tuning pass A/B'd -t 6, 8, 12 and 16 on live requests. -t 8 and -t 6 tied at roughly 74-77 tok/s, both beating -t 12 and -t 16. Neither the physical core count nor the thread count was the clean winner, which was already a warning against the folk rule of matching physical cores.

Then the model changed — from a mixture-of-experts model with layers offloaded to CPU, to a dense model that fits entirely on the GPU. Thread count went completely flat from -t 2 to -t 12. There were no longer any CPU-side layers for the threads to work on, so the setting that had been worth several percent became worth nothing, and the right value became whatever left the most cores free for other processes.

A tuning result is only valid for the workload shape it was measured on. Not the model, the shape — where the compute actually happens. Change that and your tuning notes become folklore.

The model card is a starting point, not a measurement

Both candidate quants advertised 95K and 128K context. Neither held on a 16GB card; the real ceiling, found by bisection, was 76,000, with a hard performance cliff immediately above it.

The winning quant's own card also recommended a batch and micro-batch size of 384. Measured on this hardware, that recommendation halved generation speed and quartered prompt processing against -b 2048 -ub 512.

None of this means the card was dishonest. It means a quant's author measured on their hardware and you have different hardware. Treat every published number as a hypothesis with a free test attached.

A quality win can be worth a measured speed loss

Earlier, on a 12B model, the choice was between a Q4 and a Q6 quant of the same weights:

Q6 is a 28% generation regression and leaves little room for the speech-to-text and text-to-speech processes sharing the card. Every number on that table argues for Q4.

Q6 was kept, after using both, because its answers were perceptibly more detailed. That is not a rigorous measurement and it is not presented as one — but it is the judgement the first pass exists to allow. A process that ranks purely on the numbers cannot make this call at all, which is precisely why the numbers should not be the only pass.

There may be more than one winner

The most useful outcome of a quality pass was discovering there was no single answer.

A 20B mixture-of-experts model measured roughly six times the generation speed of the 27B dense model, with about 2GB of VRAM to spare instead of 230MB. On throughput it is not close.

It still did not replace the 27B for agentic work, where the dense model produced better results. It became the pick for retrieval and voice, where latency dominates and the reasoning demands are lighter. Same card, same endpoint, two different winners for two different jobs.

This only surfaces if the quality pass is run against the workload you actually have. A single aggregate score would have picked one and quietly made the other job worse.

Quality failures are task-shaped

When quality problems do appear, they are rarely uniform, and the first report of one is often wrong about what it is.

A model was reported as "failing word counting." Fifteen tasks failed to reproduce it: a 44-word passage counted correctly, exact-length constraints from 7 to 25 words all hit exactly, hyphenated words and contractions handled correctly.

The real weakness was next door. Asked for five fruits containing no letter "a", it answered "blackberry." Exclusion constraints, not counting. Raising the reasoning effort fixed every case, at a cost of a few seconds on hard turns and no change to generation speed.

Two things worth taking from that. A vague quality complaint is a starting point for a search, not a finding. And a model that fails one narrow class of task is not a bad model — it is a model with a boundary you now know how to route around.

Two ways to get a wrong number

Both of these produced misleading measurements before they were understood.

Contention. Prompt-processing figures taken while another consumer was holding the same single-slot server were meaningless, and one latency probe was inflated to 28-37 seconds purely by cache thrash from a second client. Stop everything else before you measure.

Mismatched cache types. If the key and value cache quantisation types do not match, prompt processing collapses by five to eight times, and nothing reports an error — flash attention silently drops off its fast path. A configuration that looks reasonable and benchmarks terribly is worth checking here before anywhere else.

The method, compressed

1. Two passes. Quality decides the file; speed and context tune it afterwards.
2. Pass two does not reopen pass one.
3. Run the quality pass on the work you actually do, not on a synthetic benchmark.
4. Treat model-card numbers — context, batch size, throughput — as hypotheses with a free test attached.
5. A tuning result belongs to the workload shape it was measured on. Re-measure after any change to where the compute happens.
6. Accept a measured speed loss for a quality gain when you have used both. Write down that you did it on purpose.
7. Expect more than one winner if you have more than one kind of job.
8. Stop other clients before benchmarking, and check that your cache types match.