Aiodic

Back to the table

6 Beginner Ll Language Model Core Concepts

Language Model

A model trained on huge amounts of text to predict language.

What it is

A language model is a computer program that has learned how language tends to go, so it can predict what comes next in a piece of text. Give it "The cat sat on the" and it will tell you that "mat", "floor", and "sofa" are likely next words, and that "volcano" is not. That single skill, predicting the next piece of text, turns out to be the engine behind every AI chatbot you have used.

When people talk about ChatGPT, Claude, or Gemini, they usually mean a large language model, or LLM for short. "Large" refers to two things: the enormous amount of text the model learned from (a meaningful slice of the public internet, plus books, code, and other writing) and the enormous size of the model itself, measured in parameters, the billions of adjustable numbers that store what it has learned. On this table the element is called simply "Language Model" to fit the box, but in practice today almost every language model you meet is a large one.

Here is the basic loop, stripped to its bones. You type a message. The model reads it, along with any instructions the company has added behind the scenes, and calculates which small piece of text is most likely to come next. Those small pieces are called tokens: often a whole short word, sometimes part of a longer word, a punctuation mark, or a space. The model picks one token, adds it to the end, and then does the whole calculation again with that new token included. Then again, and again, hundreds of times, until the answer is complete. When you watch a chatbot's reply appear word by word on your screen, you are watching that loop happen.

It is worth sitting with how strange that is. There is no separate "understanding" step followed by a "writing" step. The model doesn't draft an outline, look facts up in a database, and then compose a reply. It predicts, one token at a time, and everything it seems to do, from writing a poem to fixing a spreadsheet formula, emerges from that prediction.

So how does predicting the next word become something that can explain tax rules or write working code? The answer is the training. A modern LLM is built in stages:

The design that nearly all of today's LLMs use is called the Transformer, introduced by Google researchers in a 2017 paper titled "Attention Is All You Need". Its key trick, called attention, lets the model weigh every earlier word in the text when predicting the next one, instead of reading strictly left to right and forgetting as it goes. That made it possible to train much bigger models on much more text, and the rest of this article is, in a sense, what happened next.

Why it matters

It's the part of AI most people actually touch. When someone says "I asked AI", they almost always mean they typed into a product built on a large language model. Chatbots, writing assistants, email summaries, customer-service bots, coding helpers, and the search boxes that now answer in full sentences are all language models underneath, wrapped in different interfaces. Understanding the engine helps you understand every one of those products at once.

It explains both the magic and the mistakes. Once you know that the model is predicting plausible text, a lot of confusing behaviour makes sense. It can write fluently about almost anything, because it has seen writing about almost everything. It can also state something false with complete confidence, because a fluent, plausible sentence is exactly what it was trained to produce, whether or not it is true. That failure has its own name, hallucination, and it is a direct consequence of how language models work, not a bug that one company forgot to fix.

It explains why the way you ask matters so much. Because the model continues whatever text it is given, your prompt is not just a question; it is the start of a document the model will try to complete sensibly. A vague prompt gets the most average continuation. A prompt with context, an example, and a clear format gets something much closer to what you wanted. Almost every prompting technique on this table, from few-shot examples to setting a role, works by changing what "the likely next text" is.

It's general-purpose. Earlier AI systems were usually built for one job: one model for translation, another for spam filtering, another for sorting reviews. A large language model handles all of those and more with the same underlying model, simply because you describe the job in words. For a small business or a solo creator, that is the real revolution. You don't need to commission a custom AI system for each task; you need to describe the task well.

It has limits you need to plan around. A language model only knows what was in its training text, which stops at a certain date (its knowledge cutoff) unless the product also lets it search the web or read your documents, a technique known as grounding. It can only take in so much text at once, a limit called the context window. And because it picks each token with a little randomness, controlled by a setting called temperature, the same question can get different answers. None of these limits make LLMs useless. They make them tools you use with your eyes open.

Real-world examples

Claude Shannon's guessing game (1951)

Long before computers could write, the mathematician Claude Shannon, the founder of information theory, was already treating English as something you could predict. In a paper titled "Prediction and Entropy of Printed English", published in the Bell System Technical Journal in January 1951, he described an experiment that is essentially a human language model.

He took passages from a book, Dumas Malone's biography Jefferson the Virginian, showed a person the text up to a certain point, and asked them to guess the next letter. If the guess was wrong, they guessed again, and he recorded how many tries each letter took. In one version of the test, the person could see the previous 100 letters. People turned out to be very good at this game, because English is highly predictable once you have enough context: after "the Declaration of Indepen", nobody needs many guesses. From those results Shannon estimated how much genuinely new information each letter of English carries, and his estimate was low: between about 0.6 and 1.3 bits per letter, far less than the letters would carry if they were chosen at random.

The idea at the heart of that paper, that language is predictable and that measuring how well you can predict it tells you something deep, is the same idea behind every LLM. The difference is that today's models play Shannon's game with tokens instead of letters, against far more text than any person could read, and they never get tired.

GPT-2, the model that was held back (2019)

On 14 February 2019, OpenAI announced GPT-2, a language model with 1.5 billion parameters trained on about 40 gigabytes of text taken from roughly 8 million web pages. Its only training objective was to predict the next word. Yet the samples OpenAI published showed it writing surprisingly coherent paragraphs of news-style prose, fiction, and argument from a single opening line.

What made headlines was what OpenAI did next. Citing concerns that the model could be used to mass-produce fake news, impersonate people, or automate spam, it declined to release the full model. It released a much smaller version instead and promised to release larger ones in stages while it studied the risks. The decision split the research community. Some praised the caution; others argued the danger was overstated and that withholding the model mostly generated publicity.

OpenAI released progressively larger versions over the following months, and on 5 November 2019 it released the full 1.5-billion-parameter model, saying it had so far seen no strong evidence of misuse. Whatever you think of the decision, GPT-2 was the moment the wider world noticed that a model trained only to predict the next word could produce text that read as if a person had written it. By today's standards it is tiny: the Transformer Explainer tool listed in the resources below runs the smallest GPT-2 in an ordinary web browser.

GPT-3 and the discovery that you could just ask (2020)

On 28 May 2020, OpenAI researchers published a paper with a title that summed up its finding: "Language Models are Few-Shot Learners". It introduced GPT-3, with 175 billion parameters, which the paper described as ten times more than any previous language model of its kind.

The surprise wasn't just that GPT-3 wrote better. It was that it could perform tasks it had never been specifically trained for, if you simply showed it a few examples in the prompt. One figure in the paper shows the idea using translation: a line saying "Translate English to French", then a couple of example pairs such as "sea otter => loutre de mer", then "cheese =>" with the answer left blank. The model fills in the blank. No retraining, no special translation system, just a well-built prompt. The authors called this few-shot learning, and it is the ancestor of the few-shot prompting technique people use every day.

The paper also reported something unsettling: human evaluators had difficulty telling news articles written by GPT-3 apart from articles written by people, and the authors discussed the risks that raised. Two and a half years later, in November 2022, OpenAI put a descendant of this model family in front of the public as ChatGPT. What changed in between was mostly the next example.

A small model that people preferred to a giant one (2022)

GPT-3 was powerful but awkward to use. It continued text rather than following instructions, and it could easily drift into making things up or saying something unpleasant. In a paper published in March 2022, "Training language models to follow instructions with human feedback", OpenAI researchers described a fix. They collected examples of good answers written by people, then had people rank the model's answers from best to worst, and trained the model on those preferences. They called the result InstructGPT.

The headline result was striking. In the researchers' human evaluations, people preferred the answers of a 1.3-billion-parameter InstructGPT model to those of the original 175-billion-parameter GPT-3, even though the smaller model had over 100 times fewer parameters. Size mattered less than training the model to do what people actually asked. This recipe, pretraining followed by instruction tuning and learning from human feedback, is the approach behind ChatGPT and, in various forms, most of the chatbots that followed.

A different study published the same month made a related point from another angle. Researchers at DeepMind reported that their 70-billion-parameter model, Chinchilla, trained on about 1.4 trillion tokens, outperformed their own 280-billion-parameter model, Gopher, as well as GPT-3, while using the same amount of computing power as Gopher. Their conclusion was that many large models of the time had been trained on too little text for their size. Taken together, the two papers taught the industry that "bigger" is only one ingredient. How much text a model learns from, and how it is taught to behave, matter just as much.

An everyday case: the florist's first week with a chatbot

Imagine Dana, who runs a small flower shop and starts using a chatbot to help with her admin. On Monday she asks it to write a product description for a spring bouquet, and it produces something lovely in seconds. On Tuesday she asks it to reply to a customer complaint, and the draft is polite and sensible. On Wednesday she asks it which flowers are poisonous to cats, and it gives her a confident list. On Thursday she asks what her shop's delivery charge is, and it invents a number.

None of this is inconsistent once you know what a language model is. The product description and the complaint reply are exactly the kind of text the model has seen countless times, so its predictions are excellent. The list about cats is also the kind of thing it has read many times, but it is safety information, so Dana checks it against a veterinary source before posting it, and finds one flower missing. The delivery charge was never in its training text at all, so the model produced the most plausible-sounding answer instead of saying "I don't know". On Friday Dana starts pasting her price list into the chat before asking pricing questions. The answers get right, not because the model got smarter, but because the right text was now in front of it.

Try it yourself

You need any free chatbot: ChatGPT, Claude, Gemini, or similar. Features and plan limits change often, so check each product's current terms yourself. Allow about thirty minutes.

  1. Watch the prediction happen. Open the Transformer Explainer (linked in the resources below). Type a short sentence such as "The best thing about Mondays is" and look at the list of candidate next words with their probabilities. Try moving the temperature slider and watch the choice become more or less adventurous. This is the loop from the "What it is" section, running in front of you on a small real model.
  2. See the "continue the text" instinct. In your chatbot, paste the following and see what comes back. Notice how the format of your prompt steers the format of the answer.
Here are three product names and taglines for a candle shop:
1. Morning Fog: "Soft, slow, and grey in the best way."
2. Orchard Walk: "Apples, cold air, and a long scarf."
3.
  1. Ask the same question twice. Start a new chat and ask "Give me a name for a bakery in one word." Then start another new chat and ask exactly the same thing. You will probably get different answers. That is the model sampling from its predictions rather than retrieving one stored answer.
  2. Find the edge of its knowledge. Ask about something recent: "What happened in the news in my country last week?" Depending on the product, it will either search the web, say it doesn't know, or answer from older training data. Then ask it directly: "What is your knowledge cutoff, and can you search the web in this chat?" Compare its answer with what the product's own help pages say.
  3. Catch a confident guess. Ask about something very specific and obscure that you know well, such as the history of your street, a small local business, or a detail from your own work. Check every claim. Then paste in a reliable source (a web page, a document you wrote) and ask again with this prompt:
Using only the text below, answer my question. If the answer is not in the text, say "Not in the text".

[paste your text here]

Question: [your question]

Notice how much more accurate the answer is when the facts are in front of it. This is the idea behind grounding and RAG.

  1. Ask it to explain itself, then judge the explanation. Ask: "Explain in plain English how you produced your last answer." The model will give a sensible-sounding description, but remember it is predicting what a good explanation looks like, not reading out a log of its internal workings. Treat it as a useful story, not a report.

Common mistakes and misunderstandings

Thinking it looks things up. A language model on its own doesn't consult a database of facts. It generates text from patterns it absorbed during training. Many products now add web search or document reading on top, which helps a lot, but the model underneath is still predicting text. If an answer matters, check it.

Thinking it is just autocomplete. The opposite mistake is just as common. Yes, the core mechanism is next-token prediction, the same basic idea as the word suggestions on your phone. But to predict the next word of a legal argument, a proof, or a working program well, a model has to capture a great deal of structure about the world. "Only predicting the next word" undersells what that prediction can do, just as "only moving pixels" undersells a film.

Assuming a confident answer is a correct one. The model's tone has nothing to do with its accuracy. It will describe a real historical event and an invented one in exactly the same assured voice. Confidence is a style, not a signal.

Believing bigger is always better. Parameter counts make easy headlines, but as InstructGPT and Chinchilla showed, how a model is trained matters as much as how big it is. Smaller models are often faster, cheaper, and good enough for everyday tasks. Pick the model that does your job well, and test it on your own examples rather than trusting a size or a leaderboard.

Thinking it remembers you. The model's knowledge is fixed at training time. It doesn't learn from your conversation in the way a person would. Some products add a memory feature that saves notes about you and feeds them back in later, and some companies may use conversations to train future models depending on your settings. Check each product's current privacy settings yourself, and don't paste anything into a chatbot that you wouldn't want stored.

Treating one model as "the AI". There are many language models, made by different companies and trained in different ways, including GPT from OpenAI, Claude from Anthropic, Gemini from Google, and Llama from Meta. They have different strengths, and they change with every new version. If one gives you a poor answer, another may do better on the same task.

Assuming it will say "I don't know". Models have improved at admitting uncertainty, but the underlying pressure is always to produce a plausible continuation. You can help by telling it explicitly that "I don't know" is an acceptable answer, and by giving it the source material you want it to use.

Thinking the explanation is the reasoning. When a model explains how it arrived at an answer, it is generating a plausible explanation. That explanation is often helpful, but researchers are still working out how closely it matches what actually happened inside the model, a field called interpretability.

Related elements