How do you tell if one model is actually better than another on a benchmark?

You cannot read it off the two accuracy numbers — you have to ask whether the difference is larger than the noise. Treat each model's score as a proportion measured on N questions and put a confidence interval around it: the standard error of an accuracy near 70% on 500 questions is about 2 percentage points, so a 0.4-point gap sits deep inside the overlap of the two intervals and is indistinguishable from luck. The honest test for two accuracies on the same eval set is a two-proportion test (or, better, a paired/McNemar test that uses per-question agreement), which turns the gap and the sample size into a p-value. As a rule of thumb, detecting a 1-point difference around 70% at 95% confidence needs on the order of tens of thousands of questions — far more than most public benchmarks contain — so a small win on a small benchmark is usually not a win at all.

What is data contamination in LLM evaluation and how do you detect it?

Contamination is when the benchmark you are testing on has leaked into the model's training data, so the score measures memorization instead of capability. It inflates results silently and is the single most common way an eval lies. Detection heuristics: check for verbatim n-gram overlap between the test set and the training corpus; test the model on data created after its training cutoff (a genuine capability should transfer, a memorized benchmark will not); compare performance on a public benchmark against a freshly written, equivalent-difficulty private set (a large gap is a red flag); and probe for memorization directly — ask the model to complete a canary string or reproduce a test example it should never have seen. No single check is decisive, so strong evaluations lean on held-out, recently created, or private sets rather than trusting a public leaderboard.

Why do LLM-as-a-judge evaluations have position and length bias?

Because the judge is a language model, and language models have systematic preferences unrelated to answer quality. Position bias means the judge favors whichever response is shown first (or second) regardless of content — the fix is to run each comparison twice with the order swapped and only count agreements. Length/verbosity bias means the judge tends to prefer longer, more elaborate answers even when a short one is correct, which is the same failure mode as a reward model that gets hacked by verbosity in RLHF. Self-enhancement bias means a model-judge scores outputs from its own family higher. The MT-Bench study catalogs exactly these — "position, verbosity, and self-enhancement biases, as well as limited reasoning ability" — so an honest LLM-judge eval swaps positions, controls for length, uses a different model family as judge, and validates the judge against human labels before trusting it.

How large does an eval set need to be to detect a small accuracy difference?

It scales with the inverse square of the difference you want to detect. The standard error of an accuracy p on N questions is sqrt(p(1-p)/N), and to call a gap significant it must be roughly twice the standard error of the difference. Around 70% accuracy, detecting a 5-point gap needs a few hundred questions, a 2-point gap needs a few thousand, and a 1-point gap needs on the order of 15,000-20,000 — while a 0.4-point gap needs roughly a hundred thousand. This is why most single-benchmark leaderboard movements of a point or less are noise: the benchmark simply is not big enough to resolve them. The interactive explorer on this page computes the actual confidence intervals and p-value as you dial the gap and the sample size.

What is the difference between loss, benchmark, and human evaluation?

They sit at different rungs of an eval hierarchy, each measuring something the others cannot. Loss (or perplexity) measures how well the model predicts the next token; it is free to compute and tracks training progress and scaling, but a lower loss does not guarantee a more useful or more correct model. A static benchmark (MMLU, GSM8K) measures accuracy on a fixed set of questions — cheap and reproducible, but vulnerable to contamination, format sensitivity, and the fact that the benchmark distribution is not your task. Human or pairwise evaluation (preference judgments, an arena) captures holistic quality and real preferences that no automatic metric sees, but it is slow, expensive, and noisy. Production A/B testing measures the outcome you actually care about, but you learn it late and it is confounded by everything else in the system. Strong evaluation uses the cheap rungs to iterate fast and the expensive rungs to confirm what actually matters.

Evals & Experimental Design

Most ML claims die at one question: compared to what? This is the research-taste page — the difference between "my method gets 71.2%" and "my method is actually better." We walk the claim-evidence gap, the tuned-baseline rule, the eval hierarchy from loss to production, and benchmark contamination — then the part everyone skips: is 71.2% really better than 70.8%? Dial an interactive significance explorer with real binomial confidence intervals and a live bootstrap, watch the variance a single model has run-to-run, judge the LLM judges honestly, and design the experiment before you run it.

Concept · AI / ML.

A free, interactive, animated visual explainer of Evals & Experimental Design — built to be understood, not skimmed.

Questions

How do you tell if one model is actually better than another on a benchmark?: You cannot read it off the two accuracy numbers — you have to ask whether the difference is larger than the noise. Treat each model's score as a proportion measured on N questions and put a confidence interval around it: the standard error of an accuracy near 70% on 500 questions is about 2 percentage points, so a 0.4-point gap sits deep inside the overlap of the two intervals and is indistinguishable from luck. The honest test for two accuracies on the same eval set is a two-proportion test (or, better, a paired/McNemar test that uses per-question agreement), which turns the gap and the sample size into a p-value. As a rule of thumb, detecting a 1-point difference around 70% at 95% confidence needs on the order of tens of thousands of questions — far more than most public benchmarks contain — so a small win on a small benchmark is usually not a win at all.
What is data contamination in LLM evaluation and how do you detect it?: Contamination is when the benchmark you are testing on has leaked into the model's training data, so the score measures memorization instead of capability. It inflates results silently and is the single most common way an eval lies. Detection heuristics: check for verbatim n-gram overlap between the test set and the training corpus; test the model on data created after its training cutoff (a genuine capability should transfer, a memorized benchmark will not); compare performance on a public benchmark against a freshly written, equivalent-difficulty private set (a large gap is a red flag); and probe for memorization directly — ask the model to complete a canary string or reproduce a test example it should never have seen. No single check is decisive, so strong evaluations lean on held-out, recently created, or private sets rather than trusting a public leaderboard.
Why do LLM-as-a-judge evaluations have position and length bias?: Because the judge is a language model, and language models have systematic preferences unrelated to answer quality. Position bias means the judge favors whichever response is shown first (or second) regardless of content — the fix is to run each comparison twice with the order swapped and only count agreements. Length/verbosity bias means the judge tends to prefer longer, more elaborate answers even when a short one is correct, which is the same failure mode as a reward model that gets hacked by verbosity in RLHF. Self-enhancement bias means a model-judge scores outputs from its own family higher. The MT-Bench study catalogs exactly these — "position, verbosity, and self-enhancement biases, as well as limited reasoning ability" — so an honest LLM-judge eval swaps positions, controls for length, uses a different model family as judge, and validates the judge against human labels before trusting it.
How large does an eval set need to be to detect a small accuracy difference?: It scales with the inverse square of the difference you want to detect. The standard error of an accuracy p on N questions is sqrt(p(1-p)/N), and to call a gap significant it must be roughly twice the standard error of the difference. Around 70% accuracy, detecting a 5-point gap needs a few hundred questions, a 2-point gap needs a few thousand, and a 1-point gap needs on the order of 15,000-20,000 — while a 0.4-point gap needs roughly a hundred thousand. This is why most single-benchmark leaderboard movements of a point or less are noise: the benchmark simply is not big enough to resolve them. The interactive explorer on this page computes the actual confidence intervals and p-value as you dial the gap and the sample size.
What is the difference between loss, benchmark, and human evaluation?: They sit at different rungs of an eval hierarchy, each measuring something the others cannot. Loss (or perplexity) measures how well the model predicts the next token; it is free to compute and tracks training progress and scaling, but a lower loss does not guarantee a more useful or more correct model. A static benchmark (MMLU, GSM8K) measures accuracy on a fixed set of questions — cheap and reproducible, but vulnerable to contamination, format sensitivity, and the fact that the benchmark distribution is not your task. Human or pairwise evaluation (preference judgments, an arena) captures holistic quality and real preferences that no automatic metric sees, but it is slow, expensive, and noisy. Production A/B testing measures the outcome you actually care about, but you learn it late and it is confounded by everything else in the system. Strong evaluation uses the cheap rungs to iterate fast and the expensive rungs to confirm what actually matters.

Evals & Experimental Design

Questions

Related explainers