What it is
An embedding is a list of numbers that stands for the meaning of a piece of text. A word, a sentence, a paragraph, or a whole document goes into a program called an embedding model, and out comes a list of a few hundred or a few thousand numbers. On its own, that list means nothing to a person. Its value is in comparison: two pieces of text that mean similar things get lists that are close together, and two pieces of text that mean different things get lists that are far apart.
The easiest way to picture this is a map. Every place on Earth can be described with two numbers, its latitude and longitude. Those two numbers don't tell you anything about Paris by themselves, but they let you do something useful: work out which places are near each other. Lyon is closer to Paris than Tokyo is, and you can tell just by comparing the numbers, without knowing anything else about the cities.
An embedding does the same for meaning. Imagine a map where every word and every sentence has a spot. "Dog" sits near "puppy", "hound" and "pet". "Invoice" sits near "bill", "payment" and "receipt". "I'd like my money back" sits right next to "Can I get a refund?", even though the two sentences share almost no words. The spot on the map is the embedding, and the numbers in the list are its coordinates.
The difference from a real map is the number of directions. A paper map has two: north-south and east-west. A meaning map needs far more, because there are many ways two words can be alike or different. They can be similar in topic, in tone, in formality, in whether they describe a person or an object, in whether they're about the past or the future. Instead of two coordinates, an embedding might have 300, or 1,536, or 3,072. Nobody can picture a space with 1,536 directions, and you don't need to. The idea is exactly the same as on a flat map: each thing gets a position, and closeness means similarity. (The technical word for a list of numbers used as a position like this is a "vector", which is why you'll see the phrase "vector embeddings" and hear about "vector search".)
Where do the numbers come from? Nobody writes them by hand. An embedding model is a neural network, a program that learns patterns from examples, and it learns its positions by reading enormous amounts of text. The guiding principle is old and simple: words that turn up in similar surroundings tend to mean similar things. "Coffee" and "tea" both appear after "a cup of" and before "with milk", so the model learns to put them near each other. After reading billions of sentences, the model has placed every word, and later every sentence, somewhere sensible on its map. Nobody told it what "coffee" means. It worked out where coffee belongs from the company the word keeps.
Embeddings show up in two places in modern AI, and it helps to keep them apart:
- Inside a language model. When you type into a chatbot, your words are first split into tokens, and each token is swapped for its embedding before the model does anything else. These internal embeddings are how a language model carries meaning through its layers. You never see them.
- As a separate tool for search and matching. Companies also offer embedding models on their own, whose only job is to turn text into lists of numbers you can store and compare. This is the kind that powers "search by meaning", recommendations, and the document lookup behind RAG. This article is mostly about this second kind, because it's the one you can use.
Why it matters
It lets computers search by meaning instead of spelling. Ordinary search, the kind in most websites, email programs and shop catalogues, matches words. If you search for "cheap flights" and the page says "budget airfares", keyword search can miss it. Semantic search (search based on meaning rather than matching words) turns your question into an embedding, turns every document into an embedding ahead of time, and returns the documents whose positions are closest to your question. It finds "budget airfares" because on the meaning map it sits right beside "cheap flights".
It's how AI tools read your documents. When a chatbot answers questions about your company handbook, your course notes, or a folder of PDFs, it almost always uses embeddings behind the scenes. The documents are cut into short passages, each passage gets an embedding, and those embeddings are stored in a vector database, a database built to find nearby points quickly. When you ask a question, your question gets an embedding too, the system pulls out the few passages closest to it, and hands them to the language model to read before it answers. That's retrieval-augmented generation, or RAG. The idea was set out in a 2020 paper led by researchers at Facebook AI Research, and it's now the standard way to give a chatbot knowledge it wasn't trained on. If the embeddings are poor, the wrong passages come back, and the chatbot answers confidently from the wrong page.
It powers far more than search. Once text has a position, many jobs become a matter of measuring distance. OpenAI's own documentation lists search, clustering (grouping similar things), recommendations, spotting unusual items, and classification (sorting into categories) as the main uses. A shop can recommend products whose descriptions sit near the ones you bought. A support team can group a thousand complaints into themes without reading each one first. A publisher can find which old articles are closest to a new one, to link them together. Embeddings also work for images, audio and code, which is how some photo apps let you search your pictures by typing a description.
It's cheap and fast compared with asking a chatbot. Turning text into an embedding is a much smaller job than generating a reply, and comparing two lists of numbers takes a computer a tiny fraction of a second. That means you can compare a question against millions of documents at once, which would be far too slow and expensive if you had a language model read every one.
It carries the good and the bad of what it learned. Because embeddings learn meaning from human writing, they also learn human assumptions, including unfair ones. If most of the text an embedding model read describes nurses as women and engineers as men, its map will reflect that. Any search or recommendation built on top can quietly repeat the pattern. This is one of the most studied examples of bias in AI, and one of the examples below.
Real-world examples
Word2vec and "king minus man plus woman" (2013)
In January 2013, a Google team, with Tomas Mikolov as lead author, published a paper describing a fast way to learn word embeddings from huge amounts of text. The method became known as word2vec, and in August 2013 Google released it as free, open-source software. Google's announcement explained that the model had learned, from reading news articles with no human supervision, that Paris relates to France in the same way Berlin relates to Germany.
A companion paper that year, by Mikolov with Wen-tau Yih and Geoffrey Zweig at Microsoft Research, reported the result that made embeddings famous. The positions of words weren't just grouped by topic; the directions between them seemed to mean something. The step from "man" to "woman" pointed roughly the same way as the step from "king" to "queen". So if you took the position of "king", subtracted "man", and added "woman", you landed very close to "queen". The paper described exactly that: "King – Man + Woman" gave a result very close to "Queen". On its test of grammatical analogies, such as "big is to bigger as small is to smaller", the method got almost 40 percent right, which was remarkable for a system nobody had taught any grammar.
Google also shared a ready-made set of embeddings learned from Google News articles: 3 million English words and phrases, each with 300 numbers. It became a standard starting point for researchers, and it's one reason "king minus man plus woman" still opens most talks on the subject. The lesson for a non-technical reader is that a machine given only raw text had worked out relationships like capital cities and gender pairs on its own, just from which words appeared near which.
There's a footnote worth knowing. The standard analogy code forbids the answer from being any of the words you typed in. In 2020, researchers Malvina Nissim, Rik van Noord and Rob van der Goot showed that when you remove that rule, many famous analogies no longer come out as neatly, often because the nearest point is simply the word you started with. The embeddings really do capture relationships, but the tidy arithmetic is partly a product of how the question was asked.
"Man is to computer programmer as woman is to homemaker?" (2016)
In July 2016, researchers from Boston University and Microsoft Research, Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama and Adam Kalai, published a paper with that question as its title. They took the Google News word2vec embeddings described above and asked the same kind of analogy question: man is to computer programmer as woman is to what? The embeddings answered "homemaker". Asked "father is to doctor as mother is to what?", they answered "nurse".
The team went further than analogies. They measured which job titles sat closest to "she" and which to "he". At the "she" end were homemaker, nurse and receptionist; at the "he" end were maestro, skipper and protégé. They then asked crowd workers in the United States, through Amazon's Mechanical Turk service, to judge whether the pairings the embeddings produced reflected gender stereotypes, and the workers agreed that many did. The paper's abstract put it bluntly: even embeddings trained on Google News articles "exhibit female/male gender stereotypes to a disturbing extent". The authors noted that one might have hoped news writing, much of it by professional journalists, would show little bias.
The paper was presented at NIPS 2016, one of the field's main conferences, and it proposed a way to "debias" embeddings by removing the gender direction from words that shouldn't have one. The story didn't end there. In 2019, Hila Gonen and Yoav Goldberg published a paper titled "Lipstick on a Pig", showing that such methods mostly hide the bias rather than remove it: the stereotyped words still cluster together and the pattern can be recovered. And the 2020 paper mentioned above argued that some of the most quoted biased analogies were exaggerated by the same "can't return an input word" rule. Both points can be true at once. Embeddings do soak up stereotypes from their text, and analogies are a crude and sometimes misleading way to show it.
For anyone building with embeddings today, the practical lesson is that "search by meaning" also means "search by the meanings people have attached in writing", and those aren't always fair.
Spotify finds podcasts by meaning (2022)
In March 2022, Spotify's engineering blog described a change to how its app searches podcast episodes. Until then, podcast search mostly relied on matching the words you typed against the words in episode titles and descriptions. The post gave an example: a search for "electric cars climate impact" returned nothing under the old approach, because no episode contained those exact words, even though plenty of episodes discussed the topic.
Spotify's fix was embeddings. It trained a model to turn both search queries and episodes (their titles, descriptions, and the show's name and description) into lists of numbers in a shared space, so that a question and a relevant episode would land near each other. The embeddings for millions of episodes were calculated in advance and stored in a search engine built for finding nearby points. When you search, only your query needs to be turned into an embedding, which happens on the spot, and the system fetches the closest episodes. The blog noted that for the electric cars search, none of the episodes returned contained all the words of the query in their title, yet they were exactly what the listener wanted.
Two details from the post are useful beyond Spotify. First, the team didn't throw away keyword search. The meaning-based results became one extra source of candidates, and a final ranking step blended them with the old keyword results. Semantic search is good at "roughly this topic" and weaker at exact names and codes, so most serious systems use both. Second, the model learned from Spotify's own search history, including sessions where someone's first search failed and a reworded search succeeded. Those pairs taught the model which differently worded searches meant the same thing.
An everyday case: the help centre that couldn't find "cancel"
Imagine Tom, who sells a monthly subscription box of craft supplies and has a help centre with about sixty articles. His site's search box matches words. Customers type things like "how do I stop my box" or "I don't want it next month", and the search shows nothing, because his article is titled "Managing your subscription". They email him instead, a dozen times a week.
He switches on a semantic search option that his help centre software offers. Behind the scenes, each of his sixty articles gets an embedding once, and every search gets one as it's typed. "How do I stop my box" now lands next to "Managing your subscription" on the meaning map, and the article comes up first. The emails drop. Two new problems show up, though. A customer searching for an order number gets articles that are vaguely about orders instead of the exact order, because an embedding captures gist, not exact strings. And a search for "pause" returns the cancellation article above the one about skipping a month, because the two are close in meaning. Tom fixes the first by keeping plain keyword matching for anything that looks like a number, and the second by rewriting the skipping article so its opening lines actually say "pause". Both fixes are the kind every team using embeddings ends up making.
Try it yourself
You don't need an account or any code for these. Allow half an hour.
- Explore a real meaning map. Open the free Embedding Projector at projector.tensorflow.org. It loads a demo set called "Word2Vec 10K": 10,000 English words, each with 200 numbers, squeezed down into a 3D cloud of dots you can spin with your mouse. It can take a minute to load.
- Look up a word's neighbours. In the search box on the right, type "piano" and click the word in the list. The panel shows the words closest to it in the original 200-number space. At the time of writing, the nearest are violin, guitar, orchestral, quartet, concerto and cello. Nobody told the model these were related. It worked it out from where the words appear.
- Try a word with two meanings. Search for "bank". The closest words are almost all about money: banks, monetary, banking, fund, currency. Scroll further and you'll find river and Thames, and place names linked to the West Bank. This older kind of embedding gives each word exactly one position, so all its meanings get mashed into one spot, pulled towards whichever meaning is most common in the text it learned from. Modern embedding models look at the whole sentence, so "river bank" and "bank loan" get different positions.
- Try words you care about. Search for words from your own field, such as "novel", "recipe", "client" or "garden". Look for surprises in the neighbours: they show what the text the model learned from associated with each word, and they sometimes reveal the kind of bias described above. Try "nurse" and "engineer" and compare.
- Play a game built on embeddings. Semantle, at semantle.com, is a free daily word game. You guess a secret word, and after each guess it tells you how close your guess is in meaning, using word2vec. Spelling doesn't help at all. Five minutes of play teaches you more about how an embedding "thinks" than any diagram, including its oddities: sometimes a word you'd call unrelated scores higher than one you'd call a synonym.
- Test a real semantic search. Pick an app you already use that searches by meaning, such as a podcast app, a photo app that lets you search your pictures by description, or a help centre that offers AI search. Search for something using words you're fairly sure aren't in the title, like "episode about why sleep matters for memory" or "photos of birthday cake". Then search for an exact name or code. You'll usually see the pattern from Tom's case: meaning search is good at gist and weaker at exact strings. Features change often, so check what your apps currently offer.
- Plan how you'd use it. If you have a pile of documents you'd like a chatbot to answer questions from, paste this into any chatbot to think it through before choosing a tool:
I want to build a search or question-answering tool over my own documents.
Here is what they are: [describe them: how many, what kind, how long].
Here are five questions a real user would ask: [list them].
Explain, in plain language for a non-technical person:
1. Which of these questions semantic search with embeddings would handle well, and which need exact keyword matching.
2. How I should split my documents into passages, and roughly how long each should be.
3. Three mistakes people commonly make with this kind of setup.
Do not recommend specific paid products unless I ask.
Common mistakes and misunderstandings
Thinking the numbers mean something one by one. People sometimes imagine that number 17 in an embedding stands for "is an animal" and number 42 for "is formal". It's rarely that tidy. The meaning is spread across all the numbers together, and only the position as a whole, compared with other positions, tells you anything.
Treating "close" as "correct". Embeddings measure how related two pieces of text are, not whether one answers the other. "The refund window is 30 days" and "The refund window is 60 days" sit almost on top of each other, because they're about the same thing. A search can happily return the outdated policy. Semantic search finds the right topic; it can't tell which version is true.
Expecting it to match exact words, names and numbers. Because embeddings capture gist, they can be surprisingly bad at order numbers, product codes, rare surnames and precise phrases. That's why Spotify kept keyword search alongside its semantic search, and why most good systems today combine the two.
Mixing embeddings from different models. Each embedding model builds its own map with its own coordinates. An embedding from one company's model can't be compared with one from another's, or even from a newer version of the same model. If you switch models, every stored document has to be turned into an embedding again.
Forgetting that embeddings carry bias. The positions come from human writing, so they inherit its stereotypes. Removing them is harder than it looks, as the "Lipstick on a Pig" paper showed. If you use embeddings to rank job applicants, recommend content, or sort customers, test the results for unfair patterns, and keep a person reviewing anything important.
Assuming RAG makes a chatbot always right. Retrieval only helps if the right passages come back. If documents are split badly, or the question is worded very differently from the text, the closest passages can be the wrong ones, and the chatbot will answer fluently from them. That's a common source of hallucination in systems that are supposed to be grounded in your own documents.
Thinking bigger documents make better embeddings. One embedding for a whole 40-page report blurs everything into a single average point, like describing a whole city by one pair of coordinates. Splitting documents into passages of a few paragraphs each almost always gives better search results, because each passage gets its own precise spot.
Confusing embeddings with the language model. Embedding models turn text into numbers; they don't write. A chatbot that answers questions about your files usually uses two models: an embedding model to find the right passages, and a language model to read them and write the reply. When results are poor, it's worth asking which of the two is at fault.
Related elements
- Tokens: the small chunks of text that each get their own embedding inside a language model.
- Language Model: the kind of AI that uses embeddings internally to carry meaning through its layers.
- Transformer: the design that updates each embedding using the words around it, which is how modern embeddings tell "river bank" from "bank loan".
- Neural Network: the kind of program that learns where to place each word or sentence on the meaning map.
- Vector Database: the specialised database that stores embeddings and finds the nearest ones quickly.
- RAG: the technique that uses embeddings to find the right passages and hand them to a chatbot.
- Grounding: keeping a chatbot's answers tied to real sources, which usually depends on good retrieval.
- Bias: the unfair patterns embeddings pick up from human writing, as the 2016 study showed.
- Unsupervised Learning: learning patterns from raw data with no labels, which is how word2vec learned its map.
- Hallucination: what can happen when retrieval brings back the wrong passages and the model answers anyway.