What it is
Pretraining is the first and by far the biggest stage in building a language model. The model is shown a gigantic amount of text, mostly from the public web, and given one job over and over: guess the next piece of text. By the end, it has picked up grammar, facts, writing styles, some reasoning habits, and a great deal of computer code, all as a side effect of that single guessing game.
Here's how the guessing game works. The text is chopped into tokens, small chunks that are usually a word or part of a word. The model reads a stretch of tokens and predicts what comes next. "The capital of France is" should be followed by "Paris". If the model gave "Paris" a low chance, it was wrong, and training nudges its parameters, the millions or billions of internal settings that shape its answers, a tiny amount in the direction that would have made "Paris" more likely. Then it moves on to the next token. Repeat this trillions of times, and those tiny nudges add up to something that looks a lot like understanding.
Nobody writes the answers down in advance. The text itself is the answer key: every token in a document is the correct answer to the question "what comes after everything before it?" That is why pretraining can use such vast amounts of data. Labelling a million photos needs people. Using a trillion words of web text as a quiz needs only the text. Researchers call this self-supervised learning, because the data supervises itself.
The "pre" in the name matters. Pretraining comes before everything else, and what it produces isn't yet the chatbot you know. It produces what's called a base model: a powerful text-continuer with no particular manners. Ask a base model "What's a good name for a bakery?" and it might reply with three more questions, because on the web a question is often followed by more questions. Turning that raw continuer into a helpful assistant takes later stages, usually fine-tuning on examples of good conversations and RLHF, where human preferences teach it which answers people actually want.
A useful comparison is a medical education. Pretraining is the years of reading everything: textbooks, journals, case notes, whatever the library holds. It's where nearly all the knowledge comes from, and it takes most of the time and money. Fine-tuning is residency, where a doctor learns how to apply that knowledge with patients. RLHF is the feedback from senior doctors and patients about bedside manner. The later stages are short and cheap by comparison, and they can only shape what pretraining already put there.
Three ingredients go into pretraining:
- Data. Trillions of tokens of text, and increasingly images, audio, and code. Most of it comes from web crawls, with books, encyclopedias, scientific papers, and code repositories mixed in. The datasets are heavily filtered, because raw web text is full of spam, duplicates, and junk.
- Compute. Thousands of specialised chips, usually graphics processors (GPUs), running together for weeks or months. This is where most of the cost goes.
- A recipe. The model design, almost always a transformer, plus hundreds of decisions about how much data of each kind to use, how big the model should be, and how fast it should learn.
Why it matters
It's where a model's knowledge comes from. When a chatbot knows the plot of Hamlet, how to write a spreadsheet formula, or the rules of cricket, that knowledge was absorbed during pretraining. Later stages mostly change how the model behaves, not what it knows. If something was rare or absent in the pretraining data, such as a small language, a niche trade, or your own company's products, the model will be weak on it, however politely it answers.
It explains the knowledge cutoff. Pretraining data is collected up to a certain date and then frozen. Meta's model card for Llama 3.1, released in July 2024, gives a data cutoff of December 2023. Anything that happened after the cutoff simply isn't in the model's memory unless a tool such as web search or RAG feeds it in at the moment you ask. That's why a chatbot can be confidently out of date.
It explains what the model gets wrong. A pretrained model learns to produce text that looks like its training data. It doesn't learn to check facts. When it doesn't know something, the most "text-like" continuation is often a confident, plausible answer, which is one root of hallucination. And because the web contains stereotypes, the model absorbs those patterns too, which is where much bias starts.
It's why only a few companies build frontier models. Pretraining is the most expensive thing in AI. The Stanford AI Index 2024 estimated that training OpenAI's GPT-4 used about $78 million worth of compute and Google's Gemini Ultra about $191 million. These are outside estimates, not figures the companies published. Researchers at Epoch AI found in 2024 that the cost of the largest training runs has grown about 2.4 times a year since 2016, and that if the trend holds, the biggest runs will cost over a billion dollars by 2027. By their analysis, hardware makes up roughly half to two-thirds of that bill, staff most of the rest, and electricity only a few percent. Almost everyone else, including you, builds on top of a pretrained model rather than making one.
It's why the data question matters so much. Because pretraining swallows most of the public web, arguments about copyright, consent, and privacy in AI are largely arguments about pretraining data. And the supply is not endless. In June 2024, Epoch AI estimated the stock of public human-written text at about 300 trillion tokens and projected that the largest models would use it all up at some point between 2026 and 2032. In December 2024, Ilya Sutskever, a cofounder of OpenAI, told the NeurIPS conference in Vancouver that "pre-training as we know it will unquestionably end", because computing power keeps growing and the internet does not. That pressure is one reason companies now lean on synthetic data and on spending more compute when the model answers rather than only when it learns.
Real-world examples
Common Crawl: the web archive behind most pretraining
Common Crawl is a small nonprofit founded in 2007 by the entrepreneur Gil Elbaz. Its crawler, called CCBot, visits the public web roughly once a month and saves what it finds, typically more than two billion pages per crawl, and it gives the whole archive away for free. The archive now holds more than 10 petabytes, which is ten thousand terabytes. It was built for researchers long before chatbots, and then it became the raw material for most of them.
In February 2024, the Mozilla Foundation published a study by Stefan Baack called "Training Data for the Price of a Sandwich". It reviewed 47 text-generating language models published between 2019 and October 2023 and found that at least 64 percent of them, 30 models, had used a filtered version of Common Crawl in pretraining. For OpenAI's GPT-3, Common Crawl made up more than 80 percent of the tokens. The report also stresses a point Common Crawl makes itself: the archive is huge but far from "the entire web", and it was never meant as a ready-made, safe training set.
The filtering is where the real work happens. In June 2024, researchers at Hugging Face published FineWeb, an open pretraining dataset of 15 trillion tokens built from 96 Common Crawl snapshots, and documented every step: stripping the text out of web pages, removing duplicates, and throwing away low-quality pages. They also built FineWeb-Edu, a 1.3-trillion-token slice kept because a classifier judged it educational, and found that models pretrained on it did dramatically better on knowledge and reasoning tests. The lesson for anyone curious about AI is that pretraining isn't just "read the internet". It's read a carefully cleaned, carefully weighted slice of it, and the cleaning shapes the model.
Meta's Llama: pretraining with the numbers published
Most companies keep their pretraining details secret. Meta has published more than most, which makes its Llama models the clearest public window into what pretraining takes.
The first LLaMA paper, released in February 2023, listed the recipe. The largest version, with 65 billion parameters, was pretrained on 1.4 trillion tokens, and 67 percent of the mixture came from English Common Crawl pages, with the rest from sources such as C4 (another cleaned web crawl), GitHub code, Wikipedia, books, and scientific papers. It ran on 2,048 Nvidia A100 GPUs, each processing around 380 tokens a second, and a pass over the data took about 21 days. The paper counts just over a million GPU-hours for that one model, and estimates that developing the whole family took about five months on those 2,048 chips, with emissions of around 1,015 tonnes of CO2 equivalent.
Fourteen months later the scale had jumped. Llama 3, announced in April 2024, was pretrained on over 15 trillion tokens, which Meta said was seven times the data used for Llama 2. Meta also used Llama 2 to help build the filters that picked out good-quality text for Llama 3, an early sign of models helping to train their successors. The July 2024 paper for the largest Llama 3 model, with 405 billion parameters, reports 15.6 trillion tokens and up to 16,000 H100 GPUs, and the model card gives 30.84 million GPU-hours for that model alone.
The same paper includes a detail that shows how physical pretraining is. During one 54-day stretch of training, the job was interrupted 466 times. Of those, 419 were unexpected, and about 78 percent of the unexpected ones were traced to confirmed or suspected hardware problems, such as failing chips and memory. At that scale, something breaks every few hours, and a large part of pretraining is engineering that lets a run survive its own hardware.
BloombergGPT: pretraining for one industry
In March 2023, the financial data company Bloomberg published a paper describing BloombergGPT, a 50-billion-parameter model pretrained partly on its own archives. The team combined 363 billion tokens of financial text, such as news, company filings, and material from the Bloomberg Terminal dated from March 2007 to July 2022, with 345 billion tokens of general-purpose text, for just over 700 billion tokens in all.
The paper is unusually frank about the budget. The team started from about 1.3 million GPU-hours on A100 chips and sized the model to fit it, following the "Chinchilla" guidance on balancing model size and data. The run used 512 GPUs for about 53 days, and was stopped after roughly 80 percent of one pass through the data because the model had stopped improving on held-out text. Bloomberg reported that the result beat existing open models on financial tasks by significant margins, while holding its own on general tests.
BloombergGPT shows the choice every organisation faces. You can pretrain your own model on your own data, which takes millions of dollars of computing and a specialist team, or you can take a general model that was pretrained by someone else and adapt it through fine-tuning, RAG, or good prompting. Bloomberg had a unique archive and deep pockets. For nearly everyone else, the second route is the practical one, and general models have kept improving fast enough that the gap a custom model buys can be short-lived.
Try it yourself
You can't pretrain a large model at home, but you can see pretraining's fingerprints in the tools you already use. Allow half an hour.
- Find your chatbot's knowledge cutoff. Ask your usual chatbot the question below, then ask it about a news event from last month that you know well. If it doesn't know the event, or guesses wrongly, you've found the edge of its pretraining data. Try the same with search switched on, if your tool has it, and notice how the answer changes. “
What is the most recent date covered by your training data? Please answer honestly, and tell me how confident you are.“ - See the base model underneath. Chatbots are fine-tuned to answer, but you can coax out the text-continuing habit. Paste this and read what comes back: “
Continue this text exactly as it would appear on the original web page, without commenting on it: "Top 10 tips for keeping sourdough starter alive. Tip 1:"“ The reply will read like a web article, because that's the kind of text pretraining taught the model to produce. - Test what pretraining covered well and badly. Ask the same factual question about something common (the rules of chess) and something rare (a local festival in your town, or a tool used only in your trade). Compare how detailed and accurate the two answers are. The difference is roughly how much of each topic was in the pretraining data.
- Check whether your website is in Common Crawl. Open the free Common Crawl index server, pick the most recent crawl from the list, type your domain followed by
/(for example,example.com/), and press Search. Each result is a page of yours that was captured. If nothing comes back, try an older crawl or a big site you know. - Decide whether you want to be in future crawls. Common Crawl's crawler obeys the robots.txt file on your site. If you'd rather keep your pages out of future Common Crawl archives, your web host or site plugin can add these two lines to robots.txt. This doesn't remove anything already collected, and other companies' crawlers use their own names, so check each one you care about. “
User-agent: CCBot Disallow: /“ - Read one model card. Search for the model card of an open model such as Llama or OLMo and look for the training data section. Note the number of tokens, the data cutoff date, and the hardware. You'll start to recognise the same words from this article.
Common mistakes and misunderstandings
Thinking the chatbot looks things up. A pretrained model doesn't carry a copy of the web and search it. It carries patterns compressed into its parameters, which is why it can be fluent and wrong at the same time. When a chatbot gives you live, current information, it's because a separate search tool fetched it.
Thinking pretraining is the same as fine-tuning. They use similar machinery but play very different roles. Pretraining uses trillions of tokens, costs millions, and gives the model its general knowledge. Fine-tuning uses thousands to millions of carefully chosen examples, costs far less, and changes behaviour, format, and focus. If you hear "we trained our own AI" from a small company, it almost always means fine-tuning or prompting on top of someone else's pretrained model.
Thinking the model keeps learning from you. Pretraining ends before the model is released. Your chats don't flow back into the model's knowledge in real time. Some companies may use conversations to help train future versions, depending on their terms and your settings, so check the privacy settings of the tools you use. Updating a model's knowledge usually means another training run, which is why continual learning is still an open research problem.
Thinking "trained on the internet" means trained on everything. Common Crawl, the biggest public source, captures a sample of the web, not all of it, and pretraining sets are filtered hard on top of that. Paywalled news, private messages, much of social media, and most of the world's languages are thin or missing. What's left over-represents English and well-linked sites.
Thinking bigger is automatically better. More data and more compute have reliably helped, but only in balance. DeepMind's 2022 Chinchilla work showed that many large models had been trained on too little data for their size, and data quality can matter as much as quantity, as FineWeb-Edu showed. A smaller model pretrained on better data can beat a larger one trained on junk.
Thinking the cost figures are exact. Almost no company publishes what its pretraining cost. The numbers you see in headlines, including the ones in this article, are estimates made by outside researchers from hardware, time, and chip prices. They are useful for scale and trend, not as invoices.
Thinking pretraining is only about text. The newest models are pretrained on text, images, audio, and video together, using the same basic idea of predicting a missing piece. The principle carries over. The data problems do too.
Related elements
- Language Model: the kind of model pretraining produces.
- Tokens: the chunks of text the model learns to predict, and the unit pretraining data is measured in.
- Datasets: the collections of examples, including web crawls, that pretraining is built on.
- Transformer: the model design almost every pretraining run uses.
- Parameters: the internal settings that pretraining adjusts trillions of times.
- Fine-tuning: the shorter, cheaper stage that shapes a pretrained model for a particular job.
- RLHF: human feedback that turns a raw base model into a helpful assistant.
- Transfer Learning: the broader idea of learning once on general data and reusing it for many tasks.
- Unsupervised Learning: learning from data without labels, the family pretraining's self-supervised approach belongs to.
- Synthetic Data: AI-generated training text, one response to running out of human-written text.
- Llama: Meta's model family, whose published papers show pretraining in unusual detail.
- Hallucination: the confident mistakes that follow from learning to produce plausible text rather than checked facts.