Aiodic

Back to the table

7 Beginner Tf Transformer Core Concepts

Transformer

The architecture behind almost every modern language model.

What it is

A transformer is a design for a neural network: a blueprint for how the pieces of an AI system are wired together. It's the design behind almost every modern language model, including the ones inside ChatGPT, Gemini, and Llama. When people say "architecture" in AI, they mean exactly this kind of blueprint, the same way an architect's plan decides where the rooms and corridors go before anyone chooses the furniture.

The design was introduced by a team of eight researchers at Google in a paper published in June 2017 called "Attention Is All You Need". The title was a bold claim, and it turned out to be right. The key idea, called attention, let a computer read a whole passage at once and work out how every word relates to every other word. That sounds modest. It changed the direction of the entire field.

To see why, it helps to know how computers read text before 2017. The leading designs, called recurrent networks, read a sentence the way you might read through a drinking straw: one word at a time, left to right, carrying a small running summary in memory. By the time the network reached the end of a long paragraph, the beginning had faded, like the message in a game of telephone. Worse, because each word had to wait for the one before it, the reading could not be split up and shared across many computer chips at once. Training was slow, and making the models bigger made it slower still.

The transformer threw the straw away. It looks at every word in the passage at the same time, and for each word it asks: which other words here matter for understanding me?

Attention, explained with a highlighter. Imagine reading the sentence "The animal didn't cross the street because it was too tired." When you reach "it", you instantly know it means the animal, not the street. You did that by glancing back and linking "it" to the word that makes sense. Change the ending to "because it was too wide" and "it" now means the street. Attention is a mechanism that does that glancing back, for every word, all at once. For each word, the model gives every other word a score for how relevant it is, like running a highlighter over the sentence, bright on the words that matter and faint on the ones that don't. Then it blends in meaning from the brightly highlighted words. After this step, "it" is no longer a vague pronoun. It carries a bit of "animal" inside it.

Many highlighters at once. A transformer doesn't use just one highlighter. It uses several side by side, called attention heads, and each one learns to look for something different. One might link pronouns to the nouns they stand for. Another might connect a verb to who is doing it. Another might track which words sit next to each other. Nobody tells each head what to look for; the patterns emerge during training. Think of a panel of editors reading the same page, each with a different specialism, then pooling their notes.

Layers, stacked many times. One round of attention is a single pass of the highlighters. A transformer repeats the process in layer after layer, each one working on the output of the last. Early layers tend to pick up simple things, such as grammar and nearby words. Later layers build on those to capture meaning, tone, and the thread of an argument. Large modern models stack many dozens of layers.

Word order. Because the transformer looks at all the words at once, it has no natural sense of which came first. "Dog bites man" and "man bites dog" contain the same words. So the design attaches a position tag to each word before the reading starts, which is how the model knows the order.

Put these parts together and you get a machine that takes in text broken into tokens (small chunks of words), turns each token into a list of numbers that captures its meaning (an embedding), and passes those through the stack of attention layers. At the very end, a chatbot's transformer does one simple-sounding thing: it predicts the most likely next token. Then it adds that token to the text and does it all again, which is how a reply appears word by word on your screen.

The original 2017 transformer had two halves: an encoder that reads the input, and a decoder that writes the output, which suited translation. Later designs often kept only one half. Google's BERT kept the reading half. OpenAI's GPT models kept the writing half, and the name says so: GPT stands for "generative pre-trained transformer".

Why it matters

It made today's chatbots possible. Every popular chatbot you've used runs on a transformer or a close descendant of it. Google's technical report on Gemini says the models "build on top of Transformer decoders". Meta's Llama papers describe the same foundation. The T in ChatGPT is the transformer. If you want to understand why AI suddenly got good at language around 2020, this is the single most important reason.

It made size pay off. Because the transformer reads every word at once, the work of training can be split across thousands of chips running in parallel. Google's announcement in August 2017 said the design needed less computing to train and sped up training by up to ten times compared with the older approaches. That meant researchers could train on vastly more text and build models with vastly more parameters, the internal settings a model learns. The surprise of the following years was that bigger transformers, trained on more text, kept getting better. That discovery, more than any single clever trick, is what produced the jump from autocomplete to ChatGPT.

It reads context, not just words. Older systems often treated a sentence as a bag of keywords. The transformer understands that "bank" in "river bank" and "bank loan" are different, that "not" flips a meaning, and that a small word like "to" can change who is travelling where. This is why modern AI handles sarcasm, instructions with conditions, and long documents far better than anything before it.

It spread far beyond text. The same design turned out to work on almost anything that can be cut into pieces. In 2020, Google researchers showed in a paper called "An Image is Worth 16×16 Words" that a transformer could recognise images by chopping them into small squares and treating each square like a word. Transformers now sit inside tools for images, speech, music, video, and computer code, and inside scientific systems such as AlphaFold 2, which you'll meet below.

It explains some of AI's limits. Knowing the design helps you understand the quirks. Attention compares every token with every other token, so the work grows much faster than the text: in the original design, doubling the length of the input roughly quadruples the attention work. That is one reason every model has a context window, a maximum amount of text it can consider at once, and why very long inputs cost more and can be handled less carefully. And because a transformer is ultimately predicting likely next words, it can produce fluent text that is simply wrong, which is the root of hallucination.

It's the shared vocabulary of AI news. Once you know what a transformer is, a lot of AI headlines become readable. "A bigger context window", "a new attention method", "a mixture of experts model" (a transformer that sends each token to only a few specialist sub-networks to save effort): these are all variations on the 2017 blueprint.

Real-world examples

The Google paper with the bold title (2017)

On 12 June 2017, eight researchers then at Google, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Ɓukasz Kaiser, and Illia Polosukhin, posted "Attention Is All You Need" online. It was presented that December at NeurIPS, one of the main annual AI conferences. The paper was about translation, not chatbots. Its test was translating news text from English into German and French, and it beat the best previous systems on both, scoring 28.4 on a standard translation measure for German and 41.8 for French. The larger model reached that French result after training for 3.5 days on eight graphics chips, a small fraction of the training cost of the best earlier models.

In a Google blog post on 31 August 2017, Jakob Uszkoreit explained why this mattered with a pair of sentences about an animal that didn't cross a street, where "it" meant the animal in one sentence and the street in the other. In French, "animal" and "street" have different grammatical genders, so a translator has to know which one "it" points to. Google reported that the transformer translated both sentences correctly, while the Google Translate system of the day did not. That is attention doing its job: linking a pronoun to the right noun by looking at the whole sentence at once.

Few papers in any field have had the reach of this one. Within a year, OpenAI had built its first GPT on the transformer, and Google had built BERT. Almost every major language model since has been a transformer at heart.

BERT teaches Google Search that small words matter (2019)

In October 2018, Google researchers led by Jacob Devlin published BERT, short for "Bidirectional Encoder Representations from Transformers". It used the reading half of the transformer and trained it on huge amounts of text by hiding words and asking the model to guess them from the words on both sides.

On 25 October 2019, Pandu Nayak, a vice president of Google Search, announced that BERT was now helping Search understand about one in ten English-language searches in the United States. His main example was the search "2019 brazil traveler to usa need a visa". Before BERT, Google returned results about Americans travelling to Brazil, because it treated the query roughly as a set of keywords and the little word "to" got lost. With BERT, Search understood that "to" set the direction of travel, and returned results for Brazilians travelling to the US. Google said this was the first time it used its own Cloud TPU chips to serve search results, a sign of how much computing a transformer demanded at the time.

For most people, this was their first contact with a transformer, and they never knew it. Their searches simply started understanding what they meant.

Google Translate switches its engine (2020)

The transformer was born as a translation system, and in 2020 it took over one of the world's most used translators. On 8 June 2020, Google Research described a major upgrade to Google Translate. The old engine, based on recurrent networks, was replaced by a hybrid: a transformer to read the source sentence, paired with a faster recurrent network to write the translation. The team reported that most of the quality gain came from the transformer reading half, and that using a transformer for the writing half as well didn't help enough to justify the extra delay.

Combined with other improvements introduced at the same time, including better training data, Google reported an average gain of 5 points on its standard translation score across more than 100 languages, and 7 points for the 50 languages with the least training data. It's a useful reminder that a transformer is rarely deployed as a pure textbook design. Real systems mix and match whatever parts work best.

AlphaFold 2 uses attention on proteins (2020 to 2021)

Attention turned out to be useful well outside language. Proteins are long chains of building blocks that fold into complicated 3D shapes, and predicting a shape from the chain had been an open problem in biology for decades. DeepMind's AlphaFold 2 was entered into the CASP14 protein-prediction contest, which ran from May to July 2020, and was far more accurate than any other entry.

In the paper describing it, published in Nature on 15 July 2021 with John Jumper as first author, the core of the system is a block called the Evoformer, which uses attention to reason about which parts of the protein chain relate to which other parts. It's the same highlighter idea: instead of asking which words matter to "it", the model asks which distant building blocks are likely to end up touching once the chain folds. AlphaFold 2 is not a copy of the 2017 transformer, but it shows how far the core idea has travelled.

An everyday case: the ambiguous email

Imagine Marcus, who runs a small cleaning company, pasting this into a chatbot: "Our client told the contractor that he would need to redo the tiling before she signs off. Draft a polite reply to him." A keyword-matching system would have little hope with "he", "him", and "she". A transformer uses attention to link "he" and "him" to the contractor and "she" to the client, and drafts a reply to the contractor about the tiling. When it gets this wrong, the cause is usually a genuinely ambiguous sentence, and the fix is the same one that works with people: name the person instead of using a pronoun.

Try it yourself

You don't need any technical skills for these. Allow about half an hour.

  1. Watch a transformer think. Open the free Transformer Explainer from Georgia Tech in a desktop browser. It runs GPT-2 small, a real (if old and tiny) transformer, right on your computer. Type a short sentence, such as "The chef tasted the soup and said it needed more", and look at the list of likely next words. Then change one word earlier in the sentence, for example "chef" to "child", and see how the guesses shift. You are watching attention at work: a word at the start changed the prediction at the end.
  2. Test the pronoun trick. In any chatbot, paste the pronoun prompt at the end of this section, which asks what "it" means in two nearly identical sentences. Then invent your own pairs, and notice how one changed word at the end flips the meaning of a word near the start.
  3. See context change meaning. Ask a chatbot to translate or explain a word that has several meanings, first alone and then inside a sentence. For example, "bat" alone, then "The bat flew out of the cave at dusk", then "He swung the bat and missed". The word is the same; attention to the surrounding words is what settles which meaning is meant.
  4. Feel the context window. Paste a long document into a chatbot, such as a policy or a long article, and ask a question about a detail near the middle. Then ask about something at the very beginning. If an answer is vague or wrong, try pasting only the relevant section and asking again. You're seeing, in practice, why very long inputs are harder for a transformer than short, focused ones.
  5. Ask the model about itself, then check. Ask a chatbot: "Explain what a transformer is to a ten-year-old, using an analogy about a classroom." Compare its answer with this page. It will probably be good, which is itself a demonstration of the design, but look for anything confidently stated that you can't confirm. That's the habit to keep for everything a chatbot tells you.

The pronoun prompt for step 2:

Sentence 1: The trophy didn't fit in the suitcase because it was too big.
Sentence 2: The trophy didn't fit in the suitcase because it was too small.
For each sentence, tell me what "it" refers to, and explain which other words told you.

Common mistakes and misunderstandings

Thinking "transformer" is a brand or a product. It's a design, freely published in 2017 and used by nearly every AI company. OpenAI, Google, Meta, and many others build on it and then add their own changes. Knowing a model is a transformer tells you its family, not its quality.

Thinking attention means the model understands like a person. Attention is a way of weighing which words relate to which. It is a powerful pattern-finding trick, not awareness. The model has no idea what an animal or a street is in the way you do. It has learned, from enormous amounts of text, which words tend to go with which. That can look like understanding, and for many practical tasks it works like understanding, but it can fail in ways a person wouldn't.

Believing the model reads everything equally well. A long context window means the text fits, not that every sentence gets equal care. Details buried in the middle of a very long input can be missed. If something matters, put it near the start or end, or ask about one section at a time.

Assuming a transformer looks things up. A transformer on its own doesn't search a database. It generates text from patterns it learned during pretraining. When a chatbot quotes current facts, it's usually because extra tools have been added around the transformer, such as web search or RAG, which feeds in documents to read.

Confusing the transformer with the whole chatbot. The transformer is the engine. A chatbot like ChatGPT also includes training that teaches it to follow instructions and be helpful (fine-tuning and RLHF), a system prompt, safety filters, and often tools. Two chatbots built on similar transformers can behave very differently.

Thinking every AI model is a transformer. Most language models are, but not everything. Many image generators, for example, are built around a different idea called a diffusion model, though many of them now use transformer parts inside. Researchers also keep testing alternatives that handle very long inputs more cheaply. The transformer is dominant, not universal, and it may not stay dominant forever.

Thinking the 2017 design is what runs today. Modern models have many refinements: cheaper forms of attention, different ways of marking word order, and designs like mixture of experts. The core idea of attention remains, but today's transformer is to the 2017 paper roughly what a modern car is to a Model T.

Related elements