Aiodic

Back to the table

14 Beginner Cw Context Window Core Concepts

Context Window

How much text a model can consider at once.

What it is

The context window is the amount of text an AI model can hold in view at one moment. Think of it as the model's desk. Everything it's working with right now has to fit on that desk: the instructions the company gave it behind the scenes, anything you've typed, any files you've attached, the whole conversation so far, and the reply it's in the middle of writing. If something isn't on the desk, the model can't see it.

The size of the desk is measured in tokens, the small chunks of text that models read and write. A token is often a short word or part of a longer word, and in English a rough rule of thumb is that 100 tokens is about 75 words. So when a company says a model has a "200,000-token context window", it means the model can consider roughly 150,000 words at once, which is about the length of two ordinary novels.

It helps to be clear about what the context window is not. It isn't what the model learned in training. A language model learns general knowledge about language and the world from a huge collection of text, and that knowledge is stored in its parameters, the millions or billions of internal numbers adjusted during training. Those are fixed once training ends. The context window is different: it's the short-term working space for one conversation or one task. When you start a new chat, the desk is cleared, and anything you told the model last time is gone unless the product has a separate memory feature that copies some of it back in.

This explains a lot of everyday AI behaviour that otherwise seems strange. The model can quote a paragraph you pasted twenty minutes ago because that paragraph is still in the window. It "forgets" your instruction from the very start of a three-hour chat because, in some products, that part of the chat has been pushed off the desk to make room. And it can happily read a 300-page report you upload, because modern windows are big enough to hold it.

A few details surprise people:

Why it matters

It decides what jobs you can hand over in one go. A small window means chopping a long document into pieces and asking about each piece separately, which loses the connections between them. A large window lets you give the model a whole contract, a book manuscript, a year of meeting notes or an entire software project, and ask questions that span all of it. For a lot of real work, that's the difference between "AI can't help with this" and "AI read it in a minute".

It explains why long chats go wrong. Most people have had the experience of a chatbot that was brilliant for the first hour and then started ignoring rules it had followed perfectly, repeating itself, or contradicting something it said earlier. Very often that's the context window at work, either because early material has been dropped or because the model is struggling to pay attention across a very long conversation. Once you know that, the fix is usually obvious: start a fresh chat and bring only what matters.

A big window isn't the same as good attention. This is the most important thing to understand, and it's the part the marketing leaves out. A model can accept a very long input and still use it unevenly. Research described below shows that models tend to use what's at the start and the end of a long input better than what's in the middle, and that reliability tends to drop as inputs get longer. Anthropic's own documentation for developers says it plainly: as the number of tokens grows, accuracy and recall degrade, which it calls "context rot". Fitting your document in the window means the model can see it, not that it will read every sentence with equal care.

It affects cost and speed. Businesses that connect to AI through an API, which lets software send text to a model directly and pay by usage, are billed by the token, and every token in the window is re-read on every turn. Stuffing a huge document into every request is slow and expensive, even when it fits. That's why techniques like RAG, which fetches only the relevant passages from a larger library and puts those in the window, remain popular even as windows grow.

It's the reason "context engineering" became a skill. Since the window is limited and attention within it is uneven, deciding what goes into it, in what order, and what gets left out, has become one of the main levers for getting good results. That craft now has its own name, context engineering, and it matters more as AI agents carry out long tasks that fill the window with tool results and notes.

Real-world examples

From a few pages to whole libraries in about five years

The growth in window sizes is one of the most dramatic trends in AI, and the milestones are well documented by the companies themselves.

When OpenAI described GPT-3 in its May 2020 paper "Language Models are Few-Shot Learners", every model in the family used a context window of 2,048 tokens, which is about 1,500 words, or a few pages. That was enough for a short conversation or a short article and not much more.

On 11 May 2023, Anthropic announced it had expanded Claude's window from 9,000 tokens to 100,000 tokens, which it described as around 75,000 words. To show what that meant, it loaded the whole text of The Great Gatsby (about 72,000 tokens) into the model, changed one line so that the narrator, Nick Carraway, was described as a software engineer working on machine learning tooling at Anthropic, and asked the model to find what was different. It found the altered line in 22 seconds. Anthropic also pointed out that an average person would need around five hours to read 100,000 tokens of text.

The race then sped up. On 6 November 2023, OpenAI announced GPT-4 Turbo with a 128,000-token window, which it said was equivalent to more than 300 pages of text. Two weeks later, on 21 November 2023, Anthropic released Claude 2.1 with 200,000 tokens, which it put at around 150,000 words or more than 500 pages. On 15 February 2024, Google introduced Gemini 1.5 Pro with a one-million-token window in limited preview, and said it had tested up to 10 million tokens in research. By 5 April 2025, Meta was describing its Llama 4 Scout model as supporting 10 million tokens.

At the time of writing, a window of hundreds of thousands to around a million tokens is common among the leading models, and the exact figure depends on the model, the product and sometimes the price plan. These numbers change quickly, so check your own tool's current limits in its help pages rather than relying on any figure here.

"Lost in the Middle": the study that showed position matters

In July 2023, a team of researchers from Stanford University, the University of California, Berkeley, and the company Samaya AI, led by Nelson F. Liu and including Percy Liang, published a paper called "Lost in the Middle: How Language Models Use Long Contexts". It was later published in the journal Transactions of the Association for Computational Linguistics.

The experiment was simple to describe. The researchers gave models a question along with a stack of documents, only one of which contained the answer, and moved that one document to different positions in the stack: first, last, or somewhere in between. If a model read everything equally well, the position shouldn't matter. It mattered a great deal. The paper reports that performance was often highest when the relevant information was at the very beginning or the very end of the input, and dropped significantly when the model had to find it in the middle. That was true even for models advertised as built for long inputs.

Plotted on a chart, the results formed a U shape: strong at the start, strong at the end, a dip in the middle. The pattern echoes something psychologists have long seen in human memory, where people remember the first and last items on a list best. The practical lesson for everyday users is direct. If there's one instruction or one fact that really matters, don't bury it on page 40 of a long paste. Put it at the start, or restate it at the end next to your question.

Models have improved since 2023, and the effect is weaker in many current systems than it was in the original tests. But the principle behind the paper, that a model's attention across a long input is not perfectly even, still holds, and the providers' own guidance reflects it (see "Try it yourself").

The needle in a haystack, and what it doesn't measure

On 8 November 2023, two days after GPT-4 Turbo launched with its 128,000-token window, an independent developer named Greg Kamradt published what he called a "needle in a haystack" test. He took a long collection of essays by the investor and writer Paul Graham (the haystack), hid one unrelated sentence somewhere inside it (the needle), and asked the model a question only that sentence could answer. The needle was: "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day." He repeated the test many times, changing both the total length of the text and how deep into it the needle was placed, and published the results as a colour-coded grid.

His results showed GPT-4 Turbo's recall slipping as the text got longer, with particular trouble when the needle sat partway into the document rather than at the start. On 21 November 2023 he ran the same test on Claude 2.1 and published those results too. The test spread quickly because the grid made a hard-to-see problem visible at a glance, and it became a standard way for companies to show off long windows. When Google announced Gemini 1.5 Pro in February 2024, for example, it reported that the model found the embedded text 99 percent of the time in inputs as long as a million tokens.

The story also taught a lesson about how much the prompt matters. In December 2023, Anthropic published a post about Claude 2.1's results on Kamradt's test. It explained that the model was often reluctant to answer from a single sentence that looked out of place in the document. When Anthropic added the phrase "Here is the most relevant sentence in the context:" to the start of the model's reply, its score on the original test rose from 27 percent to 98 percent. The information had been in the window all along; the model just needed a nudge to go and find it.

The limits of the test matter as much as its results. Finding one odd sentence is much easier than real reading, where the important detail looks like everything around it and you often need to connect several pieces. Google's own documentation for developers notes that performance is much lower when a model has to find several separate items at once. In July 2025, the vector database company Chroma published a report called "Context Rot", testing 18 models including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. It found that performance varied significantly as input length changed, even on simple tasks, and that a near-perfect needle score doesn't guarantee a model will handle long inputs reliably in practice.

An everyday scenario: the novelist's drifting details

Imagine Dana, who is writing her first mystery novel and has been using one long chat with an AI assistant for three weeks. She pasted in her character notes on day one: the detective has grey eyes, drinks her coffee black, and grew up in Leeds. Since then the chat has grown to dozens of chapters of drafts, feedback and rewrites.

Lately the assistant keeps giving the detective green eyes and has twice mentioned her Manchester childhood. Dana assumes the AI is getting worse. What's actually happening is that her character notes, which were at the very top of the conversation, are now a tiny part of an enormous chat. Depending on the product, they may have been dropped or summarised to make room, or they may still be there but buried in the middle, where the model is least reliable.

The fix takes ten minutes. Dana asks the assistant to write a one-page summary of the book so far: characters, key facts, plot points and style decisions. She checks it, corrects two mistakes, and starts a brand-new chat that opens with the corrected summary and her character sheet. She pastes only the chapter she's working on, and ends each request with a reminder: "Keep all character details consistent with the character sheet above." The drift stops, and the replies get faster too.

Try it yourself

You need any AI chatbot and a long document you're happy to share with it, such as a public report, a long article or a book chapter you've written. Allow 30 minutes.

  1. Find your tool's limit. Search your chatbot's help pages for "context window" or "conversation length". Note what it says, and whether the limit differs between the free and paid plans. At the time of writing, many chat products don't show the number prominently, and some handle long chats by summarising or dropping older messages without telling you, so it's worth knowing what yours does.
  2. Get a feel for size. Paste a long document into a free token counter, such as the tokenizer OpenAI offers on its website, and note how many tokens it uses. Compare that with your tool's limit. A 20-page report is often somewhere around 10,000 tokens, but it varies with the language and the layout, so measure rather than guess.
  3. Run your own needle test. Open a fresh chat. Take your long document and, somewhere in the middle, add one sentence that clearly doesn't belong, such as "The office cat is called Biscuit and is afraid of staplers." Paste the whole thing and ask: "What is the office cat called, and what is it afraid of?" Then try again in another fresh chat with the sentence near the very start. Notice whether the answer changes, and how confident the model sounds.
  4. Now ask a real question. Ask something that requires connecting two parts of the document, such as "Does the conclusion on the last page agree with the figures in the second section? Quote both." This is much harder than finding the cat, and it's much closer to real work. Check the quotes against the document yourself.
  5. Put the document first and the question last. When you paste a long document, put it at the top and your instructions and question at the bottom. Anthropic's guidance for its models says that putting the question at the end can improve response quality by up to 30 percent in its tests, especially with several documents, and Google's documentation for Gemini gives the same advice to put the question after the material.
  6. Ask for quotes. For anything important, ask the model to quote the exact passage it's relying on before it answers. It's a simple way to make it go and look, and it lets you check that the passage really exists, which guards against hallucination, where the model confidently makes something up.
  7. Make a handoff summary before a long chat goes stale. When a conversation has run for a long time, or the model starts forgetting things, paste this, check the result, and use it to open a new chat:
Please write a handoff summary of this conversation so I can continue in a new chat.
Include:
1. The goal of the project in one or two sentences.
2. Every rule, preference or constraint I've given you, as a bulleted list.
3. Key facts and decisions so far.
4. What we were in the middle of, and the next step.
Do not add anything that isn't in the conversation. If something is unclear, say so.
  1. Repeat the key rule at the end. In the new chat, put your most important rules at the top, and repeat the single most important one at the bottom of any long request. It feels redundant. It works, for the same reason the "lost in the middle" study found: the start and the end get the most attention.

Common mistakes and misunderstandings

Thinking the chatbot remembers you from last time. The context window is cleared when you start a new chat. If a product seems to remember you, it's because a separate memory feature is copying notes into the new window, and you can usually see and edit those notes in the settings. Without that, each chat starts from scratch.

Assuming a bigger window means better answers. A larger window means more fits. It doesn't mean the model reads it all equally well. Giving a model 500 pages when it only needs five often produces a worse answer than giving it the right five, because the useful part has to compete for attention with everything else. Trim before you paste.

Keeping one chat going forever. Long conversations collect contradictions, abandoned ideas and old instructions you've since changed. The model has to weigh all of it every time. A fresh chat with a clean summary almost always performs better than chat number 400 in the same thread.

Burying the important instruction. Instructions in the middle of a long paste are the ones most likely to be overlooked. Put the task and the key rules where they can't be missed: at the top in a short brief, and restated at the end next to the question.

Believing "it read the whole thing" because it didn't complain. When a document is too big, some tools cut it off, and some only take part of an attachment, without always telling you clearly. If the model's answer doesn't mention anything from the second half of your document, ask it directly what the last section says. If it can't tell you, it may never have seen it.

Forgetting that replies take up space. Asking for a 5,000-word answer uses up the window as surely as pasting a 5,000-word document. In long working sessions, ask for shorter replies until you need the long one.

Trusting the headline number. The advertised window is the maximum the model accepts, not a promise about quality at that length. A model can pass a needle test at a million tokens and still miss a subtle point in a 50-page contract. Test with your own documents and your own kind of question.

Confusing the context window with training data. The window holds what's in front of the model now. What it learned in training is a separate thing, frozen at a certain date. Pasting a document into the window is how you give a model information it never learned, which is the idea behind grounding, and the model's answers are only as current as what you put there.

Pasting private material without thinking. Everything in the window is sent to the company running the model. A large window makes it tempting to upload a whole client folder or a full inbox. Check the product's current data and privacy terms first, and leave out anything you wouldn't be comfortable sharing.

Related elements