How every number on a verdict is computed

Cost per successful task

Every call goes through OpenRouter and its usage.cost is summed over every attempt, including retries after invalid output or API errors. That total is divided by the tasks that passed. A cheap model that fails often or needs retries can cost more per success than a pricier one; this is the number that shows it. Attempts per task above 1.00 is the retry tax.

Scoring

Classification: the reply must be exactly one label from the dataset's list (case and punctuation normalised). JSON extraction: the reply must parse as a JSON object, and each field the source text actually states is compared with the expected value; fields the text does not state are not scored. Long free-text fields match at token F1 ≥ 0.7.

The bar and the status chips

You set the pass rate you need. Cleared: pass rate and the lower end of its 95% Wilson interval are at or above the bar. Borderline: pass rate at the bar, interval reaches below it. Below bar: pass rate under it. Errored: more than 20% of calls failed at the API.

The paired match with your current model

Both models answer the same tasks. Count b = tasks the baseline passed and the candidate failed, and c = the reverse. An exact one-sided binomial test on those discordant pairs asks whether the candidate is no more than 5 points worse than the baseline (default margin). At p < 0.05 the candidate "matches". The verdict is the cheapest model that matches and clears the bar.

Reproducing a run

Models sample at temperature 0 and are still not fully deterministic. A re-run lands within about ±5 points of pass rate and ±10% of cost. Prices change; each page shows the date of its OpenRouter price snapshot. The re-run script reads OPENROUTER_API_KEY from your environment, sends it only to openrouter.ai and never writes it to disk.

Datasets