case study
ModelVerdict — an LLM arena for your own prompts
LLMArenaEvalTypeScript
Problem, architecture and tradeoffs behind a side-by-side LLM comparison tool
the problem
Picking an LLM for a feature usually goes like this: skim a leaderboard, pick the highest-ranked model you can afford, and hope. But leaderboards measure aggregate performance on benchmark tasks — not your prompts, your tone requirements, your output format. The mismatch shows up in production.
ModelVerdict makes the comparison cheap: one prompt in, two answers out, side by side, and you pick the winner.
architecture
┌──────────┐ one prompt
│ client │ ───────────────────────┐
└──────────┘ ▼
┌───────────────┐
│ api route │
│ (next.js) │
└───┬───────┬───┘
parallel │ │ parallel
stream A ▼ ▼ stream B
┌──────────┐ ┌──────────┐
│ model A │ │ model B │
└─────┬────┘ └─────┬────┘
│ │
▼ ▼
side-by-side streaming UI → verdict
Key decisions:
- Both model calls run in parallel and stream independently. The whole product premise is that comparison must feel cheap — sequential calls would double the wait and kill the habit.
- One API route, provider-agnostic. Each side of the arena is just
{ baseURL, model, key }— OpenAI-compatible APIs (OpenAI, Mistral, Groq, local Ollama) all work through the same code path. - Sampling parameters are pinned across both sides. Early versions compared models at their default temperatures and produced misleading verdicts. Same temperature, same max tokens, same system prompt — or the comparison is noise.
tradeoffs i made
| decision | why | cost |
|---|---|---|
| stream raw text, no markdown render during streaming | fair visual comparison, no layout jumping | final answer re-renders once at the end |
| no automatic judge | human eyes are the eval — that's the point | doesn't scale past manual use |
| verdicts stored client-side only | zero backend state, deploys anywhere | no cross-device history |
what i'd do differently
- Blind mode first. Knowing which model is which biases the vote — hiding identities until after the verdict should have been v1, not the backlog.
- Persist verdicts with categories (code, writing, extraction…) so patterns emerge: "model A wins at code, model B at tone."
- Token/latency overlay — sometimes the right question isn't "which is better" but "is the cheaper one good enough."