What it is
A benchmark is a standard test that many AI models take, so their scores can be compared. It's a fixed set of questions or tasks, a fixed way of marking the answers, and a published score. When a company launches a new model and says it "scores 88 percent on GPQA" or "tops the leaderboard", that's a benchmark talking.
The idea is borrowed from school exams and from car magazines. Every student in a country sits the same exam paper, so a grade in one town means roughly the same as a grade in another. Every car goes round the same test track, so you can compare 0 to 60 times. Benchmarks do the same for AI. Without them, every company would test its model on questions it chose itself, and nobody could compare anything.
Most benchmarks have three parts:
- A question set. Hundreds or thousands of items: maths word problems, multiple-choice questions from university exams, bugs in real software, web tasks to complete.
- A marking rule. Usually automatic. For multiple choice, the answer letter matches or it doesn't. For coding, the fix passes the tests or it doesn't.
- A score. Usually the percentage right, often shown on a public leaderboard, a ranked table of models.
AI has run on benchmarks for a long time. In the 2010s, the ImageNet contest for recognising objects in photos was the scoreboard that proved deep learning worked (that story is told in Datasets). Once language models arrived, the tests moved from photos to words, and the famous names today are mostly written exams.
A few you'll see in launch announcements, in plain words:
- MMLU ("Massive Multitask Language Understanding"): about 16,000 multiple-choice questions across 57 school and university subjects, from maths and law to medicine. The general-knowledge exam.
- GSM8K: primary-school maths word problems. Tests whether a model can do several simple steps in a row.
- GPQA: very hard science questions written by PhD-level experts and designed so you can't simply look up the answer.
- Humanity's Last Exam: 2,500 expert questions across more than 100 subjects, built in 2025 because the older tests had become too easy.
- SWE-bench: real bugs from real software projects; covered in Coding Agents.
- Arena: not a fixed test at all but a site where people vote blind between two answers; covered in Evals.
A benchmark is different from an eval, although the words overlap. A benchmark is a public, general test that compares models with each other. An eval is your private test for your own task. Benchmarks tell you which models are strong in general. Only your own eval tells you which model is best at your job.
Why it matters
It's how progress gets measured. When people say AI improved enormously between 2020 and 2026, the evidence is mostly benchmark scores. A test that models once failed badly is now passed easily, and that's a real, checkable fact rather than marketing. Stanford's AI Index reported that on three hard tests introduced in 2023 (MMMU, GPQA and SWE-bench), top scores rose by 18.8, 48.9 and 67.3 percentage points respectively in a single year.
It's how you'll be sold to. Almost every model launch leads with a chart of benchmark bars, the new model's bar a little taller than its rivals'. If you choose between AI tools, pay for a subscription, or build a business on one, those charts are aimed at you. Knowing how to read them protects your money.
Benchmarks wear out. A benchmark is useful only while models find it hard. Once the best models all score above 90 percent, the test can no longer tell them apart. Researchers call this saturation. The field is on a treadmill: build a hard test, watch models climb it within a year or two, build a harder one.
Benchmarks can leak. Models learn from enormous scrapes of the internet during pretraining. Benchmark questions are published on the internet, often with the answers. If those questions end up in the training data, the model may have seen the exam paper before the exam. This is called contamination, and it can make a score look better than the model's real ability.
Benchmarks can be gamed. There's an old idea from economics called Goodhart's law, usually summed up as "when a measure becomes a target, it ceases to be a good measure." Once a leaderboard decides which models get attention and investment, companies have every reason to tune for the leaderboard rather than for the real skill. That can mean training on material that looks very like the test, testing many private versions and publishing the best, or presenting a special version that isn't the one you can use. The Alignment article shows the same pattern in AI systems that meet the goal as written while missing the goal as meant.
Benchmarks have errors too. They're written by people, quickly and at scale, and some of the "correct" answers are wrong. When a test's own answer key is unreliable, the last few points of a score mean very little.
None of this makes benchmarks useless. They're the best shared ruler the field has. It means a score is a clue, not a verdict.
Real-world examples
MMLU: from impossible exam to solved problem
In September 2020, Dan Hendrycks and colleagues at UC Berkeley and elsewhere released MMLU, "Measuring Massive Multitask Language Understanding". It collected 15,908 multiple-choice questions across 57 subjects, from elementary maths and US history to law and professional medicine. With four options per question, random guessing scores 25 percent. The smaller versions of GPT-3, then the most famous language model, scored about that. The largest GPT-3 scored 43.9 percent. The authors estimated that expert-level accuracy would be roughly 89.8 percent, and said that on every one of the 57 subjects the best models still needed substantial improvement.
That gap closed in about three years. In December 2023 Google announced that Gemini Ultra had passed 90 percent, though with a method that took many attempts per question (the details are in Self-consistency). By early 2025, the team behind Humanity's Last Exam could write that models "now achieve over 90% accuracy on popular benchmarks like MMLU". The test was saturated.
Two findings show why the last few points of MMLU meant little anyway. In June 2024, a study titled "Are We Done with MMLU?" led by Aryo Pradipta Gema at the University of Edinburgh re-checked 5,700 of the questions by hand. It estimated that about 6.5 percent of questions contained errors, and in the virology section 57 percent of the questions it analysed had errors. When a test's own answers are wrong that often, a model moving from 90 to 91 percent may simply be agreeing with more mistakes. The same month, researchers released MMLU-Pro, a harder version with ten answer options instead of four and more questions that need reasoning; it dropped model scores by 16 to 33 percentage points compared with MMLU. The treadmill started again.
GPQA and Humanity's Last Exam: building tests that can't be looked up
In November 2023, David Rein and colleagues at New York University published GPQA, "A Graduate-Level Google-Proof Q&A Benchmark": 448 multiple-choice questions in biology, physics and chemistry, written by people with or studying for PhDs. The "Google-proof" part was tested directly. Experts in the right field got 65 percent right. Highly skilled people from other fields, given unrestricted web access and spending over 30 minutes per question on average, got only 34 percent. GPT-4 got 39 percent. Within about a year, Stanford's AI Index recorded top scores on GPQA rising by nearly 49 percentage points.
So the Center for AI Safety and Scale AI went further. On 23 January 2025 they released Humanity's Last Exam, built from more than 70,000 submitted questions by nearly 1,000 contributors at over 500 institutions in 50 countries, narrowed to questions that the best models of the day got wrong. At launch, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro and OpenAI's o1 all answered fewer than 10 percent correctly. The public set now holds 2,500 questions, and a paper describing it was later published in Nature.
Then the pattern repeated, faster. At the time of writing, the independent testing firm Artificial Analysis lists the best model at 61.4 percent on Humanity's Last Exam without tools. And the exam has its own answer-key problem: in July 2025, the research lab FutureHouse reported that about 29 percent of the text-only chemistry and biology answers it examined appeared to conflict with published research. The exam's organisers partly confirmed this in their own review and committed to revising the set over time. Even a test written by a thousand experts needs checking.
GSM1k: catching contamination with a fresh exam
GSM8K, the primary-school maths test, has been public since 2021, answers and all. By 2024 many models scored very highly on it, and some researchers suspected that the questions had leaked into training data. Proving that is hard, because companies rarely say exactly what their models were trained on.
In May 2024, Hugh Zhang and colleagues at Scale AI tried a clean approach. They wrote GSM1k, 1,205 brand-new problems made to match GSM8K's style and difficulty as closely as possible, and deliberately kept it private so it couldn't leak. Then they gave both tests to many models. A model that truly learned the skill should score about the same on both. Several did not: some models scored up to 8 percent lower on the fresh questions, and the Phi and Mistral families showed signs of what the authors called systematic overfitting, meaning they did better on the familiar test than on the new one across almost all sizes. The researchers also found that models that were more likely to reproduce GSM8K questions word for word tended to show bigger drops, which hints at partial memorisation.
The same study had a reassuring finding: the frontier models from the Claude, GPT and Gemini families scored about the same on both tests, so their maths skill was real. The lesson isn't "all scores are fake". It's that a score on a famous public test is only trustworthy if the model would do as well on new questions it has never seen.
Llama 4 and the leaderboard version
On 5 April 2025, Meta released its Llama 4 models. Its announcement highlighted that an "experimental chat version" of Llama 4 Maverick scored an Elo rating of 1417 on LMArena, the voting leaderboard where people pick the better of two anonymous answers. That put it near the top.
Within days, people noticed that the experimental version wasn't the model Meta had released for download; it had been tuned for conversation, and testers reported it gave long, chatty replies full of emojis, the kind of style that wins votes. LMArena said Meta should have made clearer that it had submitted a customised model, updated its policies, and published more than 2,000 of that model's head-to-head battles so anyone could inspect them. When the arena then ranked the public release, TechCrunch reported on 11 April that it sat in 32nd place, below older models such as GPT-4o and Claude 3.5 Sonnet. A Meta spokesperson said the company experiments with all kinds of custom variants. Meta's head of generative AI separately denied claims that the company had trained on test sets.
Nobody here broke a rule in secret; the "experimental" label was printed in the announcement. That's what makes it a useful lesson. The small print on a benchmark claim can matter more than the headline number. (The wider debate about private testing on the arena, from the "Leaderboard Illusion" paper, is covered in Evals.)
An everyday case: the chart in the launch post
Imagine Marcus, who runs a small bookkeeping firm and is deciding whether to switch his team's AI subscription. A new model launches with a bar chart: it beats its two rivals on five benchmarks. He's tempted.
Then he reads the footnotes. Two of the five benchmarks are about competition maths, which his team never does. One score used 64 attempts per question with the best answer picked, while the rivals' scores are single attempts copied from their own launch posts. The rivals shown aren't their latest versions. And the one benchmark closest to his work, answering questions about long documents, shows the new model ahead by one point, which is within normal run-to-run variation. None of this is dishonest. It just means the chart answers a different question from his. He runs ten of his own real tasks through both models instead, and keeps his current subscription.
Charts deserve a sceptical eye in a more basic way too. At OpenAI's GPT-5 launch livestream in August 2025, one coding chart drew a 52.8 percent bar nearly twice as tall as a 69.1 percent bar. OpenAI's chief executive, Sam Altman, called it a "mega chart screwup". The numbers were printed on the bars, which is why people caught it. Always read the numbers, not the shapes.
Try it yourself
Two exercises: reading a real launch chart, then building a benchmark of your own. Allow an hour for both.
Part one: read a launch chart with a checklist.
- Find a recent model announcement. Search for the latest model from any major AI company, plus "announcement" or "launch blog". Open the company's own post, not a news story about it, and find the table or chart of benchmark scores.
- List the benchmarks. Write down each benchmark name. Next to each, write in plain words what it tests (search "what is [benchmark name]" if you don't know). Put a star next to any that resemble your real work.
- Check the rivals. Which other models are compared? Are they the rivals' newest versions, or older ones? Is any obvious competitor missing?
- Check the attempts. Look for phrases like "pass@1" (one try), "pass@k" or "@32" (many tries), "majority vote", "with tools", "with extended thinking" or "high compute". Are all models tested the same way?
- Check who ran it. Are the rivals' scores from the company's own tests, or copied from the rivals' posts? Then look up the same model on an independent site, such as the Epoch AI Benchmarking Hub, and see whether the numbers match.
- Check the size of the gap. A difference of one or two points on a test with a few hundred questions can easily be noise. Big claims need big gaps.
- Check the numbers against the bars. Read the printed values and make sure the tallest bar really belongs to the biggest number.
If the post is long, a chatbot can help you with the first pass. Paste the benchmark section and use this prompt, then check its answer against the page yourself:
Here is the benchmark section of an AI model announcement.
For each benchmark, tell me in one plain sentence what it tests.
Then list: which rival models are compared, how many attempts
or what settings each score used, and any footnotes that change
how the scores should be read. Flag anything where the models
were not tested the same way. Do not add facts that are not in
the text I pasted.
[paste the benchmark section and its footnotes here]
Part two: build a ten-question personal benchmark.
- Collect ten real tasks from your own work. Use things you actually ask AI to do: a tricky customer email, a spreadsheet question, a paragraph to rewrite, a fact about your industry you know the answer to. Include two or three that current models have got wrong for you before.
- Write the right answer or the pass rule for each. For factual questions, the correct answer. For writing tasks, two or three yes-or-no checks, such as "under 100 words" or "keeps the refund date".
- Save them in one document with the prompt exactly as you'll type it, so every model gets identical wording.
- Run each task on two or three models, each in a fresh chat. Paste the answers into a table.
- Mark each answer pass or fail, and total the scores. If two models are within one point of each other, run the close cases again before deciding.
- Keep the set private and re-run it whenever a new model or a new version of your tool appears. Because you never publish it, it can't leak into anyone's training data, which makes it more trustworthy for you than any public leaderboard.
A prompt to help you draft the pass rules:
I am building a small personal test for AI models based on my work
as a [your job]. Here is one task I give AI regularly:
[paste the task]
Suggest three yes-or-no checks I could use to mark an answer as
pass or fail. Each check must be something I can judge in under
ten seconds. Do not suggest vague checks like "sounds professional".
Common mistakes and misunderstandings
Treating a top score as "best at everything". A benchmark measures one kind of skill in one format. A model that leads on competition maths may be average at summarising contracts. Look for the benchmark closest to your work, then test on your own tasks.
Assuming a high score means the model understands. A score shows the model gets the answers right on that test. It doesn't prove how it got there. The GSM1k study showed that some models do noticeably worse on fresh questions of the same kind, which is what memorising the practice paper looks like.
Comparing scores that weren't measured the same way. One attempt versus the best of many, with tools versus without, extended thinking on versus off: these settings can move a score by many points. Only compare numbers from the same setup, ideally from the same independent tester.
Ignoring saturation. When the top models all score 88 to 92 percent on a test, and the test's own answer key contains errors, the ranking at the top is close to meaningless. Newer, harder tests are more informative, and they won't stay hard for long either.
Trusting a leaderboard without checking the version. The model on the leaderboard may not be the one you can use, as the Llama 4 case showed, and models are updated after launch. Check the exact name and date.
Thinking contamination is always cheating. Test questions leak into training data mostly by accident, because they're posted all over the internet. That's why careful test-makers now keep some questions private, and why the fairest scores come from fresh or hidden tests.
Forgetting that benchmarks mostly test answers, not work. Real jobs involve messy inputs, long projects and judgement calls. Agent benchmarks such as those in Computer Use and Browser Agents, and measures of how long a task a model can finish (see Agent), try to capture more of this, but none of them is your job.
Dismissing benchmarks entirely. The opposite mistake is just as common. Scores on hard, fresh tests track real progress, and a model that fails a benchmark badly is usually weak at that skill. Use benchmarks to build a shortlist, then use your own tests to choose.
Related elements
- Evals: your private test for your own task, and the other half of choosing a model well.
- Datasets: benchmarks are datasets too, and ImageNet began as both.
- Pretraining: the huge training stage where test questions can leak in and cause contamination.
- Language Model: the kind of model most of today's famous benchmarks are written for.
- Self-consistency: answering many times and voting, which lifts scores and is why "how many attempts" matters on a chart.
- Extended Thinking: letting a model reason at length, a setting that changes scores and should be named in any comparison.
- Coding Agents: home of SWE-bench, the most-watched coding benchmark, and why one lab stopped reporting it.
- Alignment: where Goodhart's law shows up as systems that hit the target but miss the point.
- Hallucination: a failure many benchmarks now try to measure, and one your personal test should include.
- Llama: the model family at the centre of the 2025 leaderboard controversy.