Aiodic

Back to the table

15 Beginner In Inference Core Concepts

Inference

Running a trained model to get an answer.

What it is

Inference is the moment an AI model is actually used. You type a question, the model works out a reply, and the reply appears on your screen. That working-out is inference. Every chatbot answer, every AI-generated image, every automatic caption on a video is the result of a model doing inference.

It helps to think of an AI model's life in two stages. The first is training: the long, expensive process where a model learns from huge amounts of data. That happens once (or once per version), in a data centre, over weeks or months, and it produces a finished model, which is a very large file of learned numbers called parameters. The second stage is inference: taking that finished model and running it, over and over, to answer real requests from real people. Training is building the factory. Inference is running the production line every day.

A cooking comparison works well here. Training is a chef spending years learning to cook. Inference is the chef making your dinner tonight. The years of learning are already paid for, but every dinner still takes time, gas, and ingredients. In the same way, a trained model doesn't answer for free. Every prompt sets thousands of computer chips to work for a moment, and that work costs electricity and money.

For a language model, inference happens one small piece at a time. The model reads your whole prompt, then predicts the next token (a token is a chunk of text, often a short word or part of a word). It adds that token to the text, reads everything again, and predicts the next one. A 500-word answer means the model has repeated that step several hundred times. This is why answers stream onto your screen word by word rather than appearing all at once, and why a long answer takes longer than a short one.

The word "inference" comes from statistics, where it means drawing a conclusion from evidence. In everyday AI talk it has a narrower, practical meaning: running a model that has already been trained. You'll see it in phrases like "inference cost" (what it costs to answer requests), "inference speed" (how fast answers arrive), and "inference chips" (hardware designed mainly for running models rather than training them).

Two speed measures come up often, and both are worth knowing:

Why it matters

It's where almost all the everyday cost of AI lives. Training a leading model makes the headlines because it costs a fortune up front (that story belongs to pretraining). But a popular model is trained once and then answers questions for months, often for hundreds of millions of people. Each answer is a small cost, and small costs multiplied by billions of requests become enormous. When an AI company sets its prices, limits how many messages you can send on a free plan, or tells you a feature is only for paid subscribers, the reason is almost always inference cost.

It explains the prices you see. When businesses pay to use an AI model through an API (a way for one piece of software to talk to another), they're usually charged per token: so much for every million tokens you send in, and a higher rate for every million the model writes back. Output tokens usually cost more because each one needs its own pass through the model. The tokens page explains the counting. The point here is that the meter is measuring inference.

It explains why some answers are fast and some are slow. A bigger model has more numbers to crunch for every token, so it's usually slower and more expensive to run than a smaller one. That's why most AI companies offer a family of models: a large, careful one and a small, quick, cheap one. Choosing between them is a trade-off between quality on one side and speed and cost on the other. Some designs, such as mixture of experts, exist mainly to make inference cheaper by using only part of a large model for each token.

Thinking longer costs more, and it often helps. Since 2024, a new kind of model has become common: the reasoning model, which works through a problem step by step before giving its final answer (see chain of thought and extended thinking). All that hidden working-out is made of tokens, and every token is inference. The industry name for this is "test-time compute", meaning computing power spent at the moment of answering rather than during training. It turned inference from a fixed cost per answer into a dial: turn it up and the model may answer hard questions better, but you pay in waiting time and money.

It has a real energy and environmental footprint. Every answer is electricity used in a data centre, plus cooling. One answer uses very little. Billions of answers a day add up, which is why AI companies and researchers argue, sometimes fiercely, about how much energy a single prompt really takes.

It decides where AI can run. Inference can happen in the cloud, on a company's servers, or on your own device. Running a small model on a phone or laptop means your data never leaves the device and the model works offline, but the model has to be small enough to fit. Running in the cloud gives you the biggest, cleverest models but sends your request over the internet. Much of the design of modern AI products comes down to deciding which requests are handled where.

Prices have been falling fast. Because companies compete hard on inference cost, the price of a given level of capability has dropped sharply year after year. A task that was too expensive to automate two years ago may be cheap enough now. If you ruled out an AI idea for your business on cost, it's worth checking again.

Real-world examples

The 280-fold price drop (2022 to 2024)

Stanford University's AI Index is an annual report that tracks the state of AI with published data. Its 2025 edition measured how much it cost to get answers from the cheapest model that could match the original ChatGPT model, GPT-3.5, on a standard knowledge test called MMLU. In November 2022 that level of performance cost about $20 per million tokens. By October 2024, Google's small Gemini-1.5-Flash-8B model matched it for about $0.07 per million tokens. The report described this as a drop of more than 280-fold.

The same report estimated that the cost of AI hardware had been falling by about 30 percent a year and its energy efficiency improving by about 40 percent a year. These are estimates built from public price lists and benchmark scores, not a precise measure of what any company pays behind the scenes. But the direction is clear, and it matters for anyone planning to use AI in a business: inference that was a luxury in 2022 became close to free for simple tasks by 2024. The catch is that the newest and most capable models are still priced much higher, so "AI is getting cheaper" is true for a fixed level of skill, not for the best model on the market.

OpenAI's o1 and the idea of paying for thinking time (2024)

On 12 September 2024, OpenAI introduced o1, a model trained to reason step by step before answering. Alongside the launch, OpenAI published charts showing that o1's performance improved both with more training and with more time spent thinking when it answered. OpenAI called the second of these test-time compute. On a hard American maths competition used to select students for the national olympiad (the 2024 AIME), OpenAI reported that its earlier model, GPT-4o, solved about 12 percent of the problems on average, while o1 solved 74 percent on a single attempt, and more when it was allowed to try many times and pick the most common answer.

The point for inference is that the extra skill doesn't come free. OpenAI's documentation explains that a reasoning model's hidden "reasoning tokens" aren't shown in the reply but are still billed as output tokens, and that on complex problems a model may generate anywhere from a few hundred to tens of thousands of them. At the time of writing, OpenAI's API price list showed o1 at $60 per million output tokens. A short visible answer can therefore carry a long, invisible, paid-for chain of thinking behind it.

The clearest demonstration came in December 2024, when the ARC Prize team, which runs a puzzle test designed to be hard for AI, reported results for OpenAI's next reasoning model, o3. In one setting o3 scored 75.7 percent on the team's semi-private test set. In a second setting that used about 172 times as much computing, it scored 87.5 percent. The ARC Prize team's own cost estimates for the high-compute run have since been revised as OpenAI's pricing became public, but by every version they ran to thousands of dollars per puzzle. More thinking bought a better score, at a price no ordinary user would pay.

DeepSeek-R1 and the day inference prices made the news (January 2025)

On 20 January 2025, the Chinese company DeepSeek released R1, a reasoning model it said performed comparably to OpenAI's o1. It published the model's weights (its trained numbers) under the permissive MIT licence, so anyone could download and run it, and it priced its own API at $2.19 per million output tokens, against the $60 OpenAI was charging for o1 at the time.

A week later, on 27 January 2025, Nvidia, the leading maker of AI chips, lost about 17 percent of its share price in a single day, wiping roughly $589 billion off its value, which Bloomberg reported as the largest one-day drop in market value in US stock-market history. Investors were worried that if capable models could be built and run far more cheaply, the world might need fewer expensive chips. Nvidia's public response made the opposite argument: it called R1 an excellent AI advancement and a perfect example of test-time scaling, and said that inference requires significant numbers of Nvidia GPUs. In other words, cheaper, thinking-heavy models would mean more inference in total, not less.

Both readings contain something true. Cheaper inference makes each answer cost less, and it also makes people use AI for many more things. For users, the lasting effect was a round of price competition among reasoning models. For the DeepSeek family, it was a turning point in how much attention it received worldwide.

How much energy does one answer take? (2023 to 2025)

For a couple of years, a figure of about 3 watt-hours per chatbot query was widely quoted in news stories. A watt-hour is a small unit of energy: a 10-watt LED bulb uses one in six minutes. Epoch AI, an independent research group, published an estimate in February 2025 suggesting that a typical query to GPT-4o used roughly 0.3 watt-hours, about ten times less. Its analysis traced the older figure to a 2023 study by researcher Alex de Vries that assumed longer answers and older, less efficient hardware. Epoch was careful to say that its own estimate could reasonably fall anywhere from about 0.1 to 4 watt-hours, and that very long inputs and reasoning models use considerably more.

In June 2025, OpenAI's chief executive Sam Altman wrote on his blog that an average ChatGPT query used about 0.34 watt-hours, without publishing the method behind the number. In August 2025, Google published a detailed methodology and reported that the median text prompt in its Gemini apps used 0.24 watt-hours, about the same as watching television for less than nine seconds, and that the energy per median prompt had fallen 33-fold between May 2024 and May 2025. Researchers quoted in the press criticised parts of Google's approach, including leaving out the water used to generate the electricity and reporting carbon emissions in a way that gave credit for Google's clean-energy purchases.

The honest summary is that nobody outside the companies can measure this precisely, the published estimates disagree, and the number depends heavily on the kind of request. A short text question sits at the low end. A long document, a reasoning model thinking hard, or a video generation sits much higher. What all the estimates agree on is that the footprint of one answer is small, and the footprint of billions of answers a day is not.

On-device inference: Apple Intelligence (2024)

On 10 June 2024, Apple described the models behind Apple Intelligence. One was a model of about 3 billion parameters designed to run directly on the iPhone, compressed so it would fit (a technique called quantization, explained on the parameters page). Apple reported that on an iPhone 15 Pro it could generate about 30 tokens per second. Requests too hard for the phone go to a larger model running on Apple's own servers, which Apple calls Private Cloud Compute.

This split is a good picture of how inference is shared out today. Quick, personal tasks run on the device, which is private and needs no connection. Heavier tasks go to the cloud. Google's small models for phones and laptops follow a similar idea, and so do open model families such as Gemma and Phi, which are built to be small enough to run on ordinary hardware.

An everyday case: the bookshop's chatbot bill

Here's an illustrative scenario. Imagine Sam, who runs an online bookshop and adds a chatbot to answer customer questions about orders and recommendations. In testing, he uses the most capable reasoning model on offer, and the answers are excellent. After the first busy month, the bill is several times what he expected. Looking at the usage report, he sees why: most questions were simple ("Where's my order?"), but the reasoning model spent hundreds of hidden thinking tokens on each one.

He changes the setup. Simple questions now go to a small, fast model, and only the tricky recommendation questions go to the reasoning model. Customers notice that answers arrive faster, and the monthly bill falls sharply. Nothing about the models changed. Sam simply matched the amount of inference to the difficulty of the job.

Try it yourself

You'll need access to one or more AI chatbots. Free plans are fine. Allow about 30 minutes. Features, model names, and plan limits change often, so check what your chosen tool offers at the time you read this.

  1. Watch inference happen. Ask a chatbot for something long: "Write a 600-word short story about a lighthouse keeper." Watch the answer appear. Notice the short pause before the first words (that's latency) and the steady flow afterwards (that's tokens per second). Every word you see is a separate step of inference.
  2. Time a short answer and a long one. Ask "What's the capital of Portugal? Answer in one word." Then ask for a detailed 500-word explanation of Portugal's history. Use a stopwatch on your phone. The long answer takes much longer, because every extra token is extra work.
  3. Compare a quick model with a thinking model. Many chatbots let you choose between a standard mode and a "thinking" or "reasoning" mode. Paste this puzzle into both: “ A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? Explain briefly. “ Time both answers. The thinking mode will usually take longer, and it may show a summary of its reasoning. That extra time is test-time compute: the model spending more inference to get a better answer. (The correct answer is 5 cents.)
  4. Try a question where thinking doesn't help. Ask both modes: "Suggest a friendly name for a golden retriever." The thinking mode will probably take longer and give an answer no better than the quick one. This is the lesson most people miss: extra thinking is worth paying for on hard problems, and wasted on easy ones.
  5. Look up real speed and price figures. Open the Artificial Analysis model comparison page (linked under additional resources). Find the charts for output speed (tokens per second), time to first token, and price per million tokens. Pick two models you've heard of and compare them. You'll usually see that the fastest and cheapest models aren't the most capable ones, and the most capable are neither fastest nor cheapest.
  6. Estimate your own footprint. Count roughly how many chatbot prompts you send in a day. Multiply by 0.3 watt-hours (the order of magnitude from the published estimates for a short text prompt). Compare the total with boiling a kettle, which uses somewhere around 100 watt-hours. Then remember that long documents, image generation, and reasoning modes sit well above that figure.
  7. Check what runs on your device. On a recent phone or laptop, look in the settings for AI features and read the help pages about where processing happens. Many devices now say which features run "on device" and which are sent to the cloud. If privacy matters to you, that's the line to look for.

Common mistakes and misunderstandings

Thinking the model learns from you during inference. Running a model doesn't change it. When you chat with an AI, its parameters stay exactly as they were after training. It can use what you said earlier in the same conversation, because that text is still in its context window, and some products add separate memory features. But the model itself isn't learning from each chat as it goes. Changing the model means training it again, for example through fine-tuning.

Assuming training is the expensive part and answering is free. Training costs are large and paid up front. Inference costs are paid on every single request, forever. For a popular product, the total spent on inference can easily exceed what was spent on training. That's why free plans have limits.

Believing more thinking always means better answers. Reasoning modes help most on maths, logic, planning, and tricky analysis. For simple facts, casual writing, or quick rewording, they mostly add waiting time and cost. Choose the mode that fits the task.

Forgetting about the hidden tokens. If you use a reasoning model through an API, you pay for the thinking tokens even though you don't see them. A short answer can carry a large bill. Check the usage figures, not just the length of the reply.

Treating a single energy figure as the truth. Numbers like 0.24 or 0.34 watt-hours are for a typical short text prompt, and they're company figures or outside estimates, each with its own assumptions. They don't cover image or video generation, very long documents, or heavy reasoning. Be sceptical of anyone, for or against AI, who quotes one number as if it settled the question.

Thinking "cheaper" means "the same as before, but cheaper." The falling prices in reports like the AI Index track a fixed level of capability. The newest top models are usually priced higher when they launch. Compare like with like before assuming you'll save money.

Assuming on-device AI is as capable as the cloud. A model small enough to run on a phone is much smaller than the largest cloud models. It's quicker and more private, but it will often give simpler answers. That's the trade, not a flaw.

Confusing speed with quality. A model that answers instantly isn't necessarily worse, and a slow one isn't necessarily better. Speed depends on the model's size, how busy the servers are, the hardware, and whether it's thinking first. Judge answers on their content, and test on your own tasks.

Related elements