Author here. I had a pile of models pulled locally and no way to answer "which of these is actually good, and what does it cost me in speed?" llama-bench gives you tok/s and nothing about output quality; lm-evaluation-harness gives you quality but isn't built around Ollama or LM Studio, which is what most people are actually running at home. So this measures both in one pass and puts them in one table.
The part that turned out to be harder than expected was deciding what the numbers mean:
tok/s sounds trivial until you pick a denominator. Prompt processing? Model load? I exclude both, use Ollama's server-side eval timing where it's reported, and fall back to timing the token stream client-side elsewhere. Those two paths aren't strictly comparable, which is why the methodology is in the README rather than a footnote.
Memory has no single honest answer. Runners that expose resident model size get reported directly; everything else gets a best-effort peak-RSS sample of the backend processes. Both are labelled as estimates rather than blended into one confident number.
Single-stream throughput hides half the picture on batching servers. A vLLM instance at concurrency 8 does far more aggregate work than the leaderboard's single-stream figure suggests, so batch throughput is a separate sweep reporting aggregate tok/s, speedup, and p95 latency. On a non-batching setup throughput stays flat while latency climbs, which is itself the useful result.
Quality grading is deterministic on purpose: exact numeric match, multiple choice, valid-JSON, regex, at temperature 0 with a fixed seed. There's an optional LLM-as-judge path for open-ended tasks but it's marked as a signal, not a score.
The honest limitation: the built-in suite is 31 tasks. That is a smoke test for "did this quantization break the model," not a leaderboard of record, and I make no contamination claims. It won't separate two models a few points apart, and it's English-only. If you need real eval rigor, use lm-evaluation-harness. You can point homebench at your own evals with a YAML task pack, which is what I'd suggest for anything you'd actually act on.
pip install homebench
homebench
Happy to hear where the methodology is wrong. That's the part I'm least sure about and the part that matters most.
Author here. I had a pile of models pulled locally and no way to answer "which of these is actually good, and what does it cost me in speed?" llama-bench gives you tok/s and nothing about output quality; lm-evaluation-harness gives you quality but isn't built around Ollama or LM Studio, which is what most people are actually running at home. So this measures both in one pass and puts them in one table.
The part that turned out to be harder than expected was deciding what the numbers mean:
tok/s sounds trivial until you pick a denominator. Prompt processing? Model load? I exclude both, use Ollama's server-side eval timing where it's reported, and fall back to timing the token stream client-side elsewhere. Those two paths aren't strictly comparable, which is why the methodology is in the README rather than a footnote.
Memory has no single honest answer. Runners that expose resident model size get reported directly; everything else gets a best-effort peak-RSS sample of the backend processes. Both are labelled as estimates rather than blended into one confident number.
Single-stream throughput hides half the picture on batching servers. A vLLM instance at concurrency 8 does far more aggregate work than the leaderboard's single-stream figure suggests, so batch throughput is a separate sweep reporting aggregate tok/s, speedup, and p95 latency. On a non-batching setup throughput stays flat while latency climbs, which is itself the useful result.
Quality grading is deterministic on purpose: exact numeric match, multiple choice, valid-JSON, regex, at temperature 0 with a fixed seed. There's an optional LLM-as-judge path for open-ended tasks but it's marked as a signal, not a score.
The honest limitation: the built-in suite is 31 tasks. That is a smoke test for "did this quantization break the model," not a leaderboard of record, and I make no contamination claims. It won't separate two models a few points apart, and it's English-only. If you need real eval rigor, use lm-evaluation-harness. You can point homebench at your own evals with a YAML task pack, which is what I'd suggest for anything you'd actually act on.
Happy to hear where the methodology is wrong. That's the part I'm least sure about and the part that matters most.