Aiodic

Back to the table

40 Intermediate Sc Self-consistency Prompting

Self-consistency

Sampling several answers and picking the most common.

What it is

Self-consistency is a simple way to make an AI model more reliable on questions that have one right answer. Instead of asking once and trusting the reply, you ask the same question several times, let the model work it out afresh each time, and then go with the answer that comes up most often. It's a vote, and the model is every voter.

Think about how you'd handle a tricky question in everyday life. If you're not sure how much tax is owed on an invoice, you might ask five friends who are good with numbers to work it out separately. If four of them say £312 and one says £290, you'd go with £312 without needing to know which friend is cleverest. The one odd answer is probably a slip: a misread number, a step skipped. Mistakes tend to scatter in different directions, while correct working tends to arrive at the same place. That idea, often called the wisdom of crowds, is the whole of self-consistency.

With an AI there's a twist. You don't need five friends, because the same model can play all five, provided each attempt is genuinely a fresh attempt. That depends on a setting called temperature, which controls how much randomness goes into the model's choice of words. A language model writes by picking one small piece of text at a time (these pieces are called tokens), and at temperature zero it nearly always picks the single likeliest next piece. Ask the same question five times at temperature zero and you'll mostly get the same answer five times, mistakes included. That's like asking one friend the same question five times: they'll repeat themselves, not check themselves. Turn the temperature up a little and each attempt takes a slightly different route through the problem. Now the attempts can disagree, and the disagreement is informative.

Self-consistency works best combined with chain of thought, where the model writes out its reasoning step by step before giving an answer. Each run produces a different chain of reasoning: one might work out the discount first, another the tax first. The recipe from the original research has three steps:

  1. Ask for step-by-step reasoning, so the model shows its working.
  2. Sample several answers with some randomness switched on, for example five, ten or forty runs.
  3. Take a majority vote on the final answers only, ignoring how each run got there.

Notice what's being counted. You don't compare the reasoning, which will be worded differently every time. You compare the final answer: a number, a yes or no, a letter from a multiple-choice list, a category. That's why the technique suits questions with a single, checkable answer and struggles with open-ended writing, where no two good answers look alike.

Why it matters

It catches one-off mistakes cheaply. Language models make a particular kind of error that people find baffling: they can solve a problem correctly nine times out of ten and then, on the tenth, drop a digit or misread "at least" as "at most". A single answer gives you no way to tell which kind of run you got. A vote across several runs makes the unlucky run visible and outvotes it. For many everyday tasks, that alone is worth the effort.

It needs no special access. Self-consistency doesn't require retraining, a bigger model, or any technical setup. The Google researchers who named it stressed that it works "off-the-shelf" with existing models and needs no extra training or human labelling. You can do it by hand in any chatbot, which is rare for a technique that produced such large gains in research.

Agreement is a confidence signal. The vote tells you more than the winning answer. If ten runs out of ten agree, you can be fairly confident. If the vote splits six to four, the model is unsure, and that's the moment to check the answer yourself or ask a human expert. The original paper noted this as a side benefit: the share of runs that agree gives a rough measure of how uncertain the model is. That makes self-consistency a cheap early-warning system for hallucination on factual and numerical questions, though, as the mistakes section explains, not a guarantee.

It's now built into the most powerful models. The idea that spending more computing power at answer time buys better answers, called test-time compute, is one of the big shifts in AI since 2024, covered in Inference and Extended Thinking. Voting across many samples is one of the simplest forms of it. When OpenAI introduced o1 in September 2024, it reported that on the 2024 AIME maths competition the model averaged 74 percent with a single attempt per problem and 83 percent when it took the consensus of 64 attempts. Model makers now publish scores like these routinely, which is why you need to know what the voting labels mean before you compare two models (see the Gemini example below).

It has a real cost. Five runs cost roughly five times as many tokens as one, and take five times as long, or need five requests running at once. On a paid API, where you pay per token, that adds up quickly. The original researchers pointed out that most of the gain arrives early: in their tests, 5 or 10 runs captured most of the benefit, and going much beyond that brought smaller and smaller improvements. For a one-off question in a chat window, the cost is a few minutes of your time. For a system answering thousands of customer questions an hour, it's a budgeting decision.

Real-world examples

Galton's ox and the "middlemost" guess (1907)

The idea behind self-consistency is more than a century older than AI. At the West of England Fat Stock and Poultry Exhibition in Plymouth in 1906, visitors paid to guess how much a fat ox would weigh once it had been slaughtered and dressed. The statistician Francis Galton collected the tickets afterwards and, on 7 March 1907, published an analysis in the journal Nature under the title "Vox Populi" ("the voice of the people"). After throwing out a few unreadable tickets, he had 787 guesses. Most came from people with no special expertise. Yet the middle guess, the one with as many guesses above it as below, was 1,207 pounds, and the dressed weight was 1,198 pounds, as Galton reported: within one percent.

Later researchers who re-examined his published figures found small slips (the true middle value should have been 1,208, and the organisers' letter gave the weight as 1,197), and when a reader asked for the plain average, Galton supplied it: 1,197 pounds. None of that changes the lesson. Individual guesses scattered widely, but their errors pulled in different directions and cancelled out. Self-consistency makes the same bet with an AI: many noisy but independent attempts, combined, beat one attempt. The word "independent" matters. Galton's visitors guessed separately; if they had copied the loudest person at the fair, the crowd would have been no wiser than that one person.

Google names the technique (2022)

In March 2022, a team at Google led by Xuezhi Wang, and including Jason Wei and Denny Zhou from the chain-of-thought research, published "Self-Consistency Improves Chain of Thought Reasoning in Language Models". Until then, the standard way to get an answer from a model using chain of thought was to take its single most likely line of reasoning. The team's argument was that a hard problem can usually be worked out in several different ways that all lead to the same correct answer, while wrong reasoning tends to wander off in different directions. So instead of one path, they sampled 40 reasoning paths per question with some randomness switched on and took a majority vote on the final answers.

The results were striking. On GSM8K, a set of primary-school maths word problems widely used to test AI reasoning, Google's large PaLM model (540 billion parameters) scored 56.5 percent using ordinary chain-of-thought prompting and 74.4 percent with self-consistency, a gain of 17.9 percentage points from nothing more than asking repeatedly and voting. Similar gains appeared on other tests of arithmetic and common-sense reasoning, and across four different models. The paper was published at ICLR, one of the main machine-learning conferences, in 2023 and became one of the most widely used prompting techniques. The authors were open about its main drawback: it costs more computation, which is why they suggested starting with 5 or 10 paths.

AlphaCode: a million programs, ten submissions (2022)

In February 2022, DeepMind (now Google DeepMind) announced AlphaCode, a system that writes computer programs to solve competitive programming puzzles, the kind set in online contests where thousands of human programmers race against the clock. The contest rules allowed only 10 submissions per problem, so AlphaCode's designers faced a self-consistency problem on a huge scale.

According to the team's paper, the system generated up to millions of candidate programs for each problem, using a relatively high temperature so the attempts would differ. It then threw out every program that failed the example tests given in the problem, which removed roughly 99 percent of them. That could still leave thousands. So the team ran the survivors on extra test inputs and grouped together programs that behaved identically, even if their code looked different. They submitted one program from each group, starting with the largest group. In effect, the programs voted with their behaviour: if many independently written programs gave the same outputs, that behaviour was more likely to be correct. In 10 recent contests on the Codeforces platform, AlphaCode ranked on average within the top 54 percent of participants, roughly the level of a median competitor. The work appeared in the journal Science in December 2022. It showed that sampling widely and then choosing by agreement works for code too, as long as there's some way to check the outputs against each other (see Code Generation).

Gemini's 90 percent and the fine print (December 2023)

On 6 December 2023, Google launched Gemini and announced that its largest version, Gemini Ultra, was the first model to score 90 percent on MMLU, a well-known test of 57 subjects from law to medicine, beating the 89.8 percent the test's authors estimated for human experts. The technical report showed how that score was produced: a method Google called "uncertainty-routed chain of thought", in which the model wrote 32 chain-of-thought answers per question and, if enough of them agreed, used the consensus answer. If the vote was too split, it fell back to a single standard answer. That's self-consistency with a confidence threshold, and it gave Gemini Ultra 90.04 percent.

Within days, commentators pointed out that under the more common setup, five worked examples and a single answer, Gemini Ultra scored 83.7 percent, below the 86.4 percent GPT-4 had reported that way. A report by the analytics firm GlobalData on 11 December summed up the debate on social media: many felt the comparison in Google's launch materials set a voting score against a single-answer score. The episode is a useful lesson for anyone reading AI announcements. Labels like "CoT@32", "maj@64" or "cons@64" mean the model had 32 or 64 attempts and a vote. That's a legitimate technique, but it's not the same as one answer, and it costs many times more to run. Always check that two scores were measured the same way before comparing them, as the Benchmarks article explains.

An everyday case: the quiz-night setter

Imagine Dev, who writes a weekly pub quiz and uses a chatbot to check his trickier maths and logic questions before he reads them out. One week he asks: "A shop marks a coat up 25 percent, then puts it in a sale at 20 percent off. Is the sale price higher, lower, or the same as the original cost?" The chatbot confidently says "lower". Something nags at him, so he opens four more fresh chats and asks the same question, adding "work through it step by step". Four of the five say "the same", with working that multiplies 1.25 by 0.8 and gets exactly 1. The odd one out had subtracted the percentages instead of multiplying them. Dev keeps the question, writes "the same" on his answer sheet, and adds the arithmetic as a note in case anyone argues. Five chats took him three minutes and saved him from reading out a wrong answer to forty people.

Try it yourself

You need any chatbot, such as ChatGPT, Claude or Gemini. The free versions are fine. Allow about fifteen minutes. Chat apps don't usually show a temperature setting, but at the time of writing they run with some randomness by default, which is what you need. Features and settings change, so check the current version yourself.

  1. Pick a question with one right answer that's easy to get wrong. Word problems, date calculations, unit conversions, and logic puzzles work well. Here's one to start with:
A bat and a ball cost $1.10 in total. The bat costs $1.00 more
than the ball. How much does the ball cost? Work through it step
by step, then give your final answer on the last line as
"ANSWER: [amount]".
  1. Ask it five times, each in a fresh chat. A new chat matters: in the same conversation the model can see its earlier answer and will tend to repeat it, which defeats the point. If your chatbot has a memory feature that carries information between chats, consider switching it off for this exercise.
  2. Tally the final answers only. Write down just the line after "ANSWER:" for each run. Don't grade the reasoning yet. Asking for the answer in a fixed format like this makes the tally easy; the same idea, applied to whole replies, is Structured Output.
  3. Look at the vote. If all five agree, you've learned the model is confident, which is useful, though not proof. If they split, read the working of the minority and majority answers side by side and find where they part ways. That's usually where the trap in the question is.
  4. Try a question from your own work. A VAT calculation, how many working days until a deadline, which of two pricing plans is cheaper for your usage. Run it five times the same way. You'll quickly get a feel for which kinds of question the model gets right every time and which make it wobble.
  5. Try an open-ended task for contrast. Ask for "a tagline for a candle shop" five times. You'll get five different taglines and nothing to vote on, which shows why plain self-consistency is for questions with one right answer.
  6. Try the one-prompt version. When you don't have time for five chats, you can ask for the attempts inside one reply. It's weaker than separate chats, because the model can see its own first attempt and may copy it, but it still catches some slips:
Solve the problem below three times, independently. For each
attempt, use a different method or starting point, and don't
look back at earlier attempts. Label them Attempt 1, 2 and 3,
each ending with a one-line final answer. Then compare the three
final answers. If they agree, give that answer. If they disagree,
explain where the attempts differ and which is correct, and say
how confident you are.

Problem: [paste your question here]
  1. For open-ended work, ask for a judge instead of a vote. Researchers at Google proposed a variant in 2023 called Universal Self-Consistency, where a model is shown several full answers and asked to pick the one most consistent with the others. You can try a hand-made version: generate three summaries of the same document in fresh chats, paste all three into a new chat, and ask:
Here are three summaries of the same document. Which one best
reflects the points that most of the summaries agree on? Pick
one, and list any claim that appears in only one summary so I
can check it against the original.

Common mistakes and misunderstandings

Voting at temperature zero. If you're using a tool or API where you can set the temperature, and you set it to zero for "reliability", your five runs will often be near-identical, and you'll be counting the same answer five times. Self-consistency needs the runs to be genuinely different. The original research used moderate settings, not extreme ones: enough randomness to vary the route, not so much that the answers turn to nonsense.

Asking again in the same chat. "Are you sure? Try again" in one conversation isn't self-consistency. The model can read its previous answer, and it will either repeat it or, if your question sounds doubtful, change it to please you. Independence is the whole point. Use fresh chats, or separate API calls.

Treating the majority as the truth. A vote removes random slips, not shared blind spots. If the model misunderstands a question the same way every time, or has learned a false fact, all five runs will agree on the wrong answer, confidently. Galton's crowd would have failed too if everyone had been told the wrong breed of ox. High agreement means the model is consistent, not that it's correct. For anything that matters, check the answer against a real source.

Using it on open-ended writing. There's no majority among five different blog introductions. For creative work, generating several options and choosing is still useful, but that's selection by taste, not a vote. If you want a quality check on a single draft, a different technique fits better: Self-critique, where the model reviews and improves its own answer.

Confusing it with self-critique. The two are often mixed up. Self-consistency asks many times and counts. Self-critique asks once, then asks the model to find flaws in what it wrote. They combine well: vote to get the likely answer, then ask the model to check the winning working for errors.

Forgetting the bill. Running every question ten times makes every question ten times as expensive. In a business system, save it for the questions where a wrong answer is costly, or use a split vote as the trigger for a second opinion, the way Gemini's uncertainty routing did. Measure whether it actually helps on your task with an eval before paying for it everywhere.

Comparing voting scores with single-answer scores. When a model announcement quotes "maj@32" or "cons@64", that's a self-consistency score. It's fine to report one, but it isn't comparable to another model's single-attempt score, and it doesn't tell you how the model will do when you ask it once in a chat window.

Assuming reasoning models make it unnecessary. Newer models with built-in thinking time are much less prone to simple slips, and for many everyday questions one answer is now enough. But they still make occasional errors, and they still vary from run to run. For a high-stakes calculation, a quick vote across a few fresh chats is still one of the cheapest checks available to anyone.

Related elements