Aiodic

Back to the table

24 Intermediate Fs Few-shot Prompting

Few-shot

Showing a few examples before the task.

What it is

Few-shot prompting means showing an AI a handful of examples of what you want before you ask it to do the real job. Instead of only describing the task ("write a product title that's short and punchy"), you give it two to five finished pairs: here is an input, here is the ideal output, here is another input, here is its ideal output. Then you give it a new input and leave the output blank. The model spots the pattern and continues it.

Each example is called a "shot", which is where the name comes from. No examples at all is zero-shot prompting, which is how most people use chatbots most of the time. One example is one-shot. A few is few-shot. Anthropic's documentation also calls it "multishot" prompting, and you'll see "examples", "demonstrations" and "exemplars" used for the same thing.

Here is the simplest possible version, a prompt for sorting customer messages:

Message: "Where is my parcel? It's been ten days."
Category: Delivery

Message: "The zip broke the first time I used it."
Category: Faulty product

Message: "Can I change the colour on my order?"
Category: Order change

Message: "I was charged twice this month."
Category:

You never told the model what the categories were or how to phrase them, but it will almost certainly answer "Billing" or something close, in the same short form, with no extra chat. The examples did the explaining.

The idea behind this has a technical name, in-context learning, and it's worth understanding in plain words. A language model learns during training, when its billions of internal settings (its parameters) are adjusted over months. After that, the settings are frozen. When you type a prompt, nothing inside the model is permanently changed. And yet, within a single conversation, it can pick up a new pattern from a few examples and apply it. That is in-context learning: learning that happens inside the prompt, lasts only as long as the conversation, and is forgotten the moment you start a new chat.

OpenAI's GPT-3 paper in 2020 made this famous, with a title that said it outright: "Language Models are Few-Shot Learners". The paper stressed that no retraining was involved; the tasks and examples were given "purely via text interaction with the model". The Language Model page tells that story in more detail.

Why do examples work so well? Because describing what you want is surprisingly hard. Words like "punchy", "friendly", "concise" or "in my style" mean different things to different people, and to a model they mean something like the average of everything it has ever read. An example removes the guesswork. It shows the length, the structure, the punctuation, the tone and the level of detail all at once. A picture is worth a thousand words, and in prompting, a good example is often worth a paragraph of instructions.

Why it matters

It's often the quickest improvement you can make to a prompt. Google's own prompt guidance for its Gemini models goes as far as recommending that you always include few-shot examples, saying prompts without them are likely to be less effective. Anthropic's guidance for Claude calls examples one of the most reliable ways to steer format, tone and structure, and suggests three to five of them. OpenAI's guide describes few-shot learning as a way to steer a model toward a new task without retraining it. Three of the biggest AI companies agree on very little about prompting, but they agree on this. (These guides are updated often, so check the current versions yourself.)

It controls format better than any instruction. If you need answers in a fixed shape, such as a one-word label, a table row, a subject line under 50 characters, or a particular JSON layout for a spreadsheet, showing the shape beats describing it. The model copies what it sees much more faithfully than it follows what it's told. This is why few-shot prompts are the backbone of most business uses of AI, where answers get pasted into another system that expects a precise format.

It's how you teach a model your voice. For creators this is the big one. Describing your writing style ("warm but not gushing, a bit wry, short paragraphs") gets you a generic imitation. Pasting in three real pieces you wrote gets you something far closer, because the examples carry hundreds of small choices you'd never think to describe: how you open, how long your sentences run, which words you never use, where you put the joke.

It's cheap and instant, unlike retraining. The heavier alternative is fine-tuning: training a copy of the model on hundreds or thousands of your examples so the pattern is baked in permanently. Fine-tuning costs money, takes time, needs clean data, and has to be redone when a better model comes out. Few-shot costs nothing but a few paragraphs of prompt, works in any chatbot, and you can change your examples in thirty seconds. The sensible order is almost always to try few-shot first and only consider fine-tuning if it clearly falls short.

It shows you what you actually want. Choosing examples forces a decision. If you can't find three outputs you'd be happy to see again, you probably haven't settled what "good" looks like yet, and no prompt will fix that.

The honest limits: examples take up space in the model's context window, its working memory for a conversation, and that space costs money in apps that charge by the token. Examples can also teach the wrong lesson, as the next sections show.

Real-world examples

The order of examples swung accuracy from chance to near the best (2021)

In February 2021, Tony Z. Zhao and Eric Wallace of UC Berkeley, with Shi Feng, Dan Klein and Sameer Singh, published a paper called "Calibrate Before Use: Improving Few-Shot Performance of Language Models". It was presented at ICML, one of the main machine-learning conferences, later that year.

They tested GPT-3 and GPT-2 on few-shot tasks such as deciding whether a movie review was positive or negative. What they found was unsettling. The choice of prompt format, which examples were used, and even the order of the same examples could move accuracy "from near chance to near state-of-the-art", in the words of their abstract. Same model, same task, same information: wildly different results depending on arrangement.

They traced the problem to three habits, which they named:

Their fix, called contextual calibration, was aimed at developers: measure the model's lean by giving it a meaningless input such as "N/A", then correct for it. They reported gains of up to 30 percentage points in accuracy. For everyday users, the lesson is simpler and doesn't need any code: balance your examples, shuffle their order, and don't let the last one accidentally set the tone.

Wrong answers in the examples barely mattered (2022)

A year later, a team led by Sewon Min, with colleagues including Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer, asked a sharper question: what is the model actually learning from the examples? Their paper, "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?", was posted in February 2022 and published at the EMNLP conference that year.

They ran a deliberately strange experiment. They took few-shot prompts for classification tasks and replaced the correct answers in the examples with random ones. A negative review might be labelled "positive", a sports headline labelled "business". You'd expect the model to learn nonsense. Instead, across 12 different models including GPT-3, the random labels "barely hurt" performance.

What did matter, they found, was that the examples showed three things: the set of possible answers (the "label space"), what typical inputs look like, and the overall format of the sequence. In other words, much of what the examples teach is shape, not substance.

This is useful to know, with one caution. A Google study in March 2023, "Larger language models do in-context learning differently", found that the biggest models do follow the answers in examples: shown deliberately flipped labels, they flipped their own answers to match, while smaller models stuck to what they already knew. So you shouldn't conclude that sloppy examples are harmless, especially with today's large models. But it explains a lot about how few-shot prompting behaves in practice. The model is extremely sensitive to the form of your examples: their length, their layout, their punctuation, the words they open with. That's why examples are so good at controlling format, and why a single oddity repeated across your examples tends to show up in every answer. (The Prompt page describes a related 2023 study on how much tiny formatting changes matter.)

Many-shot: when a few examples become hundreds (2024)

Few-shot prompting was born when models could only read a couple of thousand words at a time. By 2024, context windows had grown to hundreds of thousands of tokens and beyond, and two teams looked at what happens when "a few" becomes "a lot".

On 17 April 2024, a team of Google researchers posted "Many-Shot In-Context Learning". Using Gemini 1.5 Pro, with its one-million-token context window, they tried prompts containing hundreds or even thousands of examples. Performance kept improving well beyond the handful people usually use, and on some tasks it came close to what fine-tuning achieves. They also found that enough examples could override biases the model had picked up in training.

Two weeks earlier, on 2 April 2024, Anthropic had published the flip side. Its researchers described "many-shot jailbreaking": filling a long prompt with fake dialogues in which an AI assistant happily answers harmful questions, then asking a harmful question for real. With enough fake examples, up to 256 in their tests, the model would follow the pattern and ignore its safety training. It is exactly the same ability that makes few-shot prompting useful, pointed in the wrong direction. Anthropic said it had shared the details confidentially with other researchers and AI companies before publishing, and reported that one prompt-based defence cut the attack's success rate from 61 percent to 2 percent in one case.

The two papers together make the point of this article vividly. Models learn a great deal from examples, sometimes more than from their instructions, and that power works for whoever writes the examples. It is one reason AI companies build guardrails that don't rely on the prompt alone.

An everyday scenario: the newsletter that stopped sounding like a robot

This is an illustrative scenario, not a documented case. Priya writes a weekly newsletter about houseplants. She tried asking a chatbot to draft her intro paragraphs "in a friendly, witty tone". Every draft opened with "Hey plant lovers!" and used the word "thrive" twice.

She tried again, this time pasting in three of her own past intros, each on a different topic: one about overwatering, one about a plant she'd killed, one about a reader's question. Then she gave the new topic and asked for an intro in the same voice. The draft opened mid-story, the way hers do, and kept her habit of ending on a short, dry line. It wasn't perfect: it borrowed the name of her cat from one example, which she deleted. But it needed one edit instead of a rewrite.

Try it yourself

This exercise teaches a chatbot your writing voice with three examples. It works in any mainstream chatbot, including the free versions of ChatGPT, Claude and Gemini at the time of writing. Allow twenty minutes.

  1. Pick one kind of writing you do repeatedly. Social posts, product descriptions, newsletter intros, email replies to readers, back-cover blurbs. One kind only; mixing types confuses the pattern.
  2. Choose three real examples you're proud of. They should be your own writing, all the same kind, and similar in length to what you want back. Pick them for variety of subject: if all three are about the same topic, the model will think the topic is part of the style.
  3. Check them for accidental patterns. Do all three start with a question? End with the same sign-off? Mention the same person? The model will copy anything they share, so either vary it or keep it only if you want it in every answer.
  4. Run a zero-shot baseline first. In a new chat, paste this, filling in the brackets, and save the result:
Write a [kind of writing, e.g. newsletter intro] of about [length] about: [new topic].
Make it sound like me: [describe your style in one sentence].
  1. Now run the few-shot version. Start a fresh chat so the first attempt doesn't influence it, and paste this:
I'm going to show you three examples of my writing. Study the voice:
sentence length, how each one opens and ends, word choice, humour,
and what I leave out. Don't copy their topics, names or phrases.

<example>
[paste example 1]
</example>

<example>
[paste example 2]
</example>

<example>
[paste example 3]
</example>

Now write a new [kind of writing] of about [length] about: [new topic].
Match the voice of the examples. Use none of their specific details.

The tags around each example help the model see where each one starts and stops, and keep the examples separate from your instructions. Anthropic's guidance recommends exactly this kind of wrapping.

  1. Compare the two drafts side by side. Most people find the few-shot version is noticeably closer to their voice. Note what it got right and what it got wrong.
  2. Check for leaks. Read the new draft for anything lifted from your examples: a name, a phrase, a joke, the same closing line. If you find some, add a line such as "Do not reuse any sentence or name from the examples" and run it again.
  3. Ask the model what it noticed. In the same chat, paste this. The answer is often a sharp description of your style that you can reuse:
Describe the voice of my three examples as a list of 8 specific,
observable habits (for example, sentence length, how paragraphs open,
words I avoid). Be concrete. Don't praise the writing.
  1. Save the winning prompt. Keep your three examples and the instructions in a document so you can paste them whenever you need that kind of writing. That turns it into a reusable prompt template. If your chatbot lets you save custom instructions or projects, you can store it there, but check how your tool handles this, because features change.
  2. Swap one example and compare. Replace your weakest example with a different one and run the same topic again. If the result changes a lot, your examples are doing most of the work, which is the point, and it's worth choosing them carefully.

Common mistakes and misunderstandings

Examples that are all alike. If every example is about the same product, the same length and the same mood, the model can't tell which features are the pattern and which are coincidence. Vary the topics and cover at least one awkward case, such as an unusually short input or a tricky one. Anthropic's guidance asks for examples that are relevant, diverse and cover edge cases, and warns that too-similar examples teach unintended patterns.

Unbalanced labels. For sorting or labelling tasks, if four of your five examples have the same answer, the model will over-use that answer. That's the majority label bias Zhao and colleagues described. Give each answer roughly equal representation.

Letting the last example decide. Recency bias means the final example carries extra weight. Shuffle the order, and if one answer keeps dominating, check whether it happens to be the last example.

The model copying surface details. Examples teach everything they contain, not just what you meant. If all three end with an exclamation mark, every answer will too. If one mentions your dog, the new draft might as well. This is sometimes called overfitting to the examples, and Google's guidance notes that using too many examples can make it worse. Read your examples as a stranger would and ask: what do these have in common that I didn't intend?

Examples leaking into the answer. A close cousin: the model lifts whole phrases, names, facts or numbers from an example into a new answer where they don't belong. This is especially risky when examples contain real data, such as a customer's name or a price. Use invented or anonymised details in examples you'll reuse, and tell the model explicitly not to reuse specifics.

Inconsistent formatting between examples. If one example uses a bullet list and another uses a paragraph, or the labels are spelled differently ("Billing" in one, "billing issue" in another), the model has to guess which is right, and its guesses will vary. Google's guidance singles out consistency of structure and formatting across examples. Make them look identical in everything except content.

Putting in examples you don't actually like. A mediocre example is a mediocre target. The model will aim to match it. Only use outputs you'd be happy to see again.

Thinking more is always better. Three to five examples is the usual sweet spot for everyday use. Many more can help on some tasks, as the Google many-shot paper showed, but they cost space and money, and past a point they make the copying problems worse. Add examples one at a time and check whether each one actually improves the result. An eval, a small set of test cases you run every prompt change against, is the reliable way to tell.

Confusing few-shot with fine-tuning. Few-shot doesn't change the model. The moment you start a new chat, the examples are gone and you need to paste them again. Fine-tuning changes the model's settings permanently. Few-shot is a sticky note on the desk; fine-tuning is a training course.

Forgetting to combine it with instructions. Examples and instructions work best together. The examples show the shape; a sentence or two says what matters most and what to avoid. Few-shot also pairs well with chain of thought: if your examples show the reasoning written out step by step, the model tends to reason before answering too.

Related elements