Building ModelVerdict: Letting Two LLMs Fight It Out

Benchmarks tell you how a model performs on someone else's problems. They say very little about how it performs on yours.

That gap is why I built ModelVerdict — a small arena where you type one prompt and two models answer it side by side. No leaderboard, no aggregate score. Just your actual use case, answered twice, so you can judge with your own eyes.

The core idea

The flow is deliberately simple:

1. You write a prompt.
2. Two models (your pick) generate answers in parallel.
3. You compare and crown a winner.

Running the calls in parallel matters more than it sounds. Sequential calls double the wait, and the whole point of the tool is to make comparison cheap. If checking two models feels slower than just trusting one, nobody will check.

What I learned

Streaming changes perception. When both answers stream token by token, the comparison feels alive — you notice which model "gets it" earlier, which one rambles before the point. A finished wall of text hides all of that.

Temperature is a confounder. Early on I compared models at their default temperatures and drew completely wrong conclusions. Pinning the sampling parameters across both sides was the single biggest fix for fair comparisons.

Your eyes are a fine eval. For most product decisions you don't need an LLM-as-judge pipeline. Ten side-by-side comparisons on real prompts will tell you more than a benchmark table ever will.

What's next

I want to add blind mode — hide which model is which until you vote — and keep a local history of verdicts so patterns emerge over time. The code is on my GitHub if you want to poke around.