Aiodic

Back to the table

81 Advanced Rg RAG Tools & Protocols

RAG

Retrieval-augmented generation.

What it is

RAG stands for retrieval-augmented generation. It's a way of getting an AI chatbot to answer from your documents instead of from memory. Before the model writes a word, the system searches your material, picks out the few passages most likely to contain the answer, and places them in front of the model along with your question. The model then writes its reply from those passages, ideally pointing to where each claim came from.

The three words describe the three steps in reverse. Retrieval is the search. Augmented means the question gets extra material added to it. Generation is the model writing the answer.

Think of a good reference librarian. You ask a question. The librarian doesn't answer off the top of their head. They walk to the shelves, pull the three books most likely to help, open them at the right pages, and read the relevant paragraphs before they reply. If the library doesn't have anything on your question, a good librarian says so. RAG turns an AI model into that librarian, working in a library you supply.

Why go to the trouble? A language model on its own only knows what it absorbed during training. That knowledge stops at a cutoff date, contains nothing about your business, your price list, or your unpublished novel, and is stored as fuzzy patterns rather than exact facts. When it doesn't know, it may produce a plausible guess, which is how hallucination happens. RAG hands it the facts at the moment it needs them.

In plain words, a typical RAG system works like this:

  1. Split the documents into chunks. Your manuals, FAQs, contracts, or chapters are cut into short passages, often a few paragraphs each. Short passages are easier to find precisely than whole documents.
  2. Index the chunks. Each chunk is turned into an embedding, a list of numbers that captures its meaning, and stored in a vector database so it can be found by meaning rather than exact words. Many systems also keep an ordinary keyword index alongside, because meaning search can miss exact names and codes. This happens once, in advance, and again whenever the documents change.
  3. Find the relevant chunks. When someone asks a question, the question is turned into an embedding too, and the system pulls out the handful of chunks closest to it. Better systems then re-rank those candidates, checking more carefully which ones really answer the question.
  4. Hand them to the model. The chosen chunks are pasted into the model's context window, the text it can see at once, together with instructions such as "answer only from these passages".
  5. Answer with citations. The model writes a reply based on the passages and, in a well-built system, says which passage supports each point, so a person can check.

Chunking, embeddings, and indexing are where most of the engineering lives, and the Vector Database article covers the storage side in depth. What matters for everyone else is the shape of the idea: search first, then answer from what the search found.

RAG is a specific way of doing grounding, which is the broader habit of tying answers to sources you can check. When you paste one document into a chatbot and say "answer from this", you're grounding by hand. RAG automates the fetching, so it works when the library is thousands of documents and nobody could paste them all.

Why it matters

It lets AI answer about your world. Most useful questions in a business are about specifics: this client's contract, our returns policy, the setting of chapter twelve. No general model knows those. RAG is the standard way to give a chatbot that knowledge without retraining it, which is why it sits behind most "chat with your documents" features, company help bots, and internal search assistants.

It keeps answers current. Update the documents and the answers update too, the next time the index is refreshed. Nothing about the model has to change. A new price list on Monday means correct prices on Monday.

It makes answers checkable. Because the answer comes from specific passages, the system can show them. A reader can click through and confirm the claim, which is the difference between trusting a chatbot and verifying it.

It's usually cheaper and easier than the alternatives. There are three main ways to get your knowledge into an AI answer, and each suits a different job:

A common pattern is to combine them: fine-tuning or a system prompt for the voice, RAG for the facts.

It moves the risk to where you can manage it. A plain chatbot fails in ways you can't see. A RAG system fails in ways you can inspect: you can look at which chunks it retrieved, spot that the wrong one came back, and fix the document or the search. That makes it a system you can test and improve with evals, rather than a black box.

The "RAG is dead" debate. When Google announced Gemini 1.5 Pro on 15 February 2024, with an experimental window of one million tokens, which it described as more than 700,000 words, a wave of posts declared that retrieval was obsolete: why search when you can paste everything? The research since has been more balanced. A July 2024 study by Zhuowan Li and colleagues, presented at the EMNLP 2024 conference, compared the two head to head. It found that when cost was no object, long context generally scored better on average, but that RAG cost far less, and it proposed letting the model decide question by question which route to take. Anthropic's advice points the same way. For a few hundred pages, long context is often simplest. For a company's whole document store, updated daily, retrieval still earns its keep. The fair summary is that longer windows changed where the line falls, not whether the line exists.

Real-world examples

The paper that named it (2020)

On 22 May 2020, a team led by Patrick Lewis posted a paper called "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". The authors came from Facebook AI Research (now Meta AI), University College London, and New York University, and the paper was accepted at NeurIPS 2020, one of the main AI research conferences.

The idea was to pair a language model with a searchable index of Wikipedia. Instead of relying only on what the model had memorised, which the paper calls parametric memory, the system first retrieved relevant Wikipedia passages, the non-parametric memory, and generated its answer from them. On several open-domain question-answering tests, where questions can be about anything, it set new records. The authors also reported that its writing was more specific, more varied, and more factual than a comparable model working from memory alone. A useful side effect was that the knowledge could be updated by swapping the index rather than retraining the model.

The name stuck, somewhat to its authors' regret. In an interview published by NVIDIA in November 2023, Lewis, by then leading a RAG team at the AI company Cohere, said the team would have put more thought into the name had they known how widespread the work would become, and that nobody had come up with a better idea when it came time to write the paper. The unlovely acronym now describes a whole industry of products, and the basic pipeline in this article is still recognisably the one the paper described.

Morgan Stanley puts 100,000 documents behind a chatbot (2023)

On 14 March 2023, the day OpenAI launched GPT-4, Morgan Stanley Wealth Management announced it was one of a handful of launch organisations building on the model. Its press release said the bank was using GPT-4 to generate responses exclusively from internal Morgan Stanley content. That is RAG in a sentence.

In September 2023 the tool, called the AI @ Morgan Stanley Assistant, went fully live for the bank's financial advisors and their support staff. CNBC reported on 18 September 2023 that it gave advisors quick access to a database of about 100,000 research reports and documents. Jeff McMillan, the executive leading the project, told CNBC that the bank spent months curating the documents and having human experts test the answers before launch. He also noted a small adjustment for users: advisors had to ask questions in full sentences, as if talking to a person, rather than typing keywords as they would into a search engine.

By June 2024 the bank said 98 percent of its financial advisor teams had adopted the assistant. The figures are the bank's own. The lessons are general: the work that made it succeed was choosing and cleaning the documents and testing the answers, not the model itself.

New York City's business chatbot gives illegal advice (2024)

In October 2023, New York City launched an AI chatbot on its MyCity portal to help people starting and running businesses. It was built on Microsoft's Azure AI services and, as The Markup later reported, drew on information from more than 2,000 official NYC Business web pages. On paper this was ideal RAG: a well-defined library of authoritative government pages.

On 29 March 2024, the nonprofit newsroom The Markup published an investigation by Colin Lecher showing that the bot was confidently telling business owners to break the law. It said employers could take a cut of workers' tips, that landlords didn't have to accept Section 8 housing vouchers, and that businesses could refuse cash, all of which contradict New York law. The page carried a warning that the bot might occasionally produce incorrect content, but the answers were fluent and assured, and an ordinary user had no way to tell which ones were wrong. The city defended it as a pilot and promised improvements.

The bot stayed online for almost two more years. On 30 January 2026, The Markup reported that the new mayor, Zohran Mamdani, planned to shut it down, calling it "functionally unusable" and citing a cost of around half a million dollars. By early February 2026 the page said the beta test had ended.

The case shows that good source documents are not enough. Retrieval can bring back a passage about a related rule, the model can blend it with general knowledge, and a missing "I don't know" turns a gap into an invented answer. Testing with real questions whose right answers are known, the way Morgan Stanley did, is what catches this before the public does.

Microsoft's GraphRAG tackles the big-picture question (2024)

Ordinary RAG is good at needle-in-a-haystack questions, where the answer sits in one or two passages. It's bad at questions about a whole collection, such as "what are the main themes in these documents?", because no single chunk contains the answer.

On 13 February 2024, Jonathan Larson and Steven Truitt of Microsoft Research described an alternative called GraphRAG. Before any questions are asked, a language model reads the whole collection and builds a knowledge graph: a map of the people, places, organisations, and ideas in it, and how they connect. It then groups related items into clusters and writes a summary of each. Questions about the whole collection can be answered from those summaries.

Microsoft demonstrated it on thousands of news articles about the war in Ukraine from June 2023. Asked for the top five themes, ordinary RAG came back with scattered topics, because the question gave its search nothing specific to match. GraphRAG came back with themes that reflected what the collection was actually about. Microsoft released GraphRAG as open-source software on 2 July 2024. Its researchers were candid about the trade-off: building the graph costs a lot of model time up front, and whether that pays off depends on the job. For most small businesses, plain RAG is the right start. GraphRAG is worth knowing about because it explains a failure you may well see: ask a summary question of a RAG system and get a confident answer drawn from three random pages.

An everyday scenario: the series bible

This is an illustration, not a documented case. Dana writes a fantasy series, now five books long. Her readers notice everything, and she keeps losing track of small facts: the colour of a minor character's eyes, which city the river runs through, what year the treaty was signed. She uploads the five manuscripts and her notes file into an AI tool that answers from uploaded files.

Now, before writing a scene, she asks, "What have I said about Maren's scar, and in which book?" The tool pulls the three passages that mention it and quotes them with the chapter they come from. When she asks about a character's mother, whom she has never described, the tool says it can't find anything, and she knows she's free to invent. She still checks each quote against the manuscript before relying on it. But a job that used to mean an evening of searching now takes a minute, and the tool is a search assistant for her canon, not a co-author.

Try it yourself

You'll need an AI tool that answers questions from files you upload. Many do this with RAG behind the scenes: Google's NotebookLM (renamed Gemini Notebook in 2026), and the "projects" or file-upload features in ChatGPT, Claude, Copilot, and others. At the time of writing several offer a free tier with limits on how many files and questions you can use. Features, names, and limits change often, so check the current terms yourself, and don't upload anything confidential to a tool whose privacy terms you haven't read.

Pick a small set of documents you know well: your business FAQ and policies, a handbook, course notes, or a few chapters of your own writing. Knowing them well is the point, because you're going to test the tool.

  1. Upload your documents. Start small, three to ten files. Give them clear names with dates, such as "Returns policy, March 2026", because the tool will show those names in its citations.
  2. Set the ground rules. Many tools let you add instructions for a project or notebook. If yours does, paste the prompt below there. If not, paste it at the start of the chat.
Answer my questions using ONLY the uploaded files.
For every answer, name the file and quote the sentence or sentences you relied on, word for word.
If the files do not contain the answer, reply exactly: "Not in the files." Do not guess, and do not fill gaps from general knowledge.
If two files disagree, say so and quote both.
  1. Ask a question that IS in the files. Choose one where you know the exact answer, such as "How many days do customers have to return an item?" Check the quote against the original file. Does the sentence exist, and does it actually say what the answer claims?
  2. Ask a question that ISN'T in the files. Choose something plausible that your documents never mention, such as "Do you offer gift wrapping?" This is the most important test. A well-behaved system says "Not in the files." A poorly behaved one invents a friendly, reasonable answer. If that happens, you've just seen the MyCity failure in miniature.
  3. Ask a question that needs two files. For example, "If an overseas customer returns a sale item, what do they pay?" when shipping rules and sale rules live in different documents. See whether the answer combines them correctly and cites both, or whether it answers from one and ignores the other.
  4. Ask a big-picture question. Try "What are the three most common topics across all these files?" Compare the answer with what you know. Summary questions are where simple retrieval struggles, so notice whether the answer feels like it comes from the whole set or from a few random pages.
  5. Test a stale fact. Upload an older version of one document alongside the new one and ask a question where they differ. Does it pick the current one, flag the conflict, or quietly use the old one? This is why clear names with dates matter.
  6. Write down what you found. Keep your test questions and the right answers in a note. Every time you add or change documents, run the same questions again. That list is the start of an eval, and it's how businesses keep RAG systems honest.

A follow-up prompt for checking any answer:

For your last answer, list each claim you made as a separate line.
Next to each claim, give the file name and the exact quote that supports it.
If any claim is not directly supported by a quote, mark it "UNSUPPORTED".

If your tests go well and you run a small business, the natural next step is a customer-facing FAQ bot built the same way. Before letting it talk to customers, run your saved questions through it, including the ones it shouldn't be able to answer, and keep a person checking its replies at first.

Common mistakes and misunderstandings

"It's using my documents, so it can't be wrong." RAG lowers the rate of made-up answers. It doesn't remove it. The model can misread a passage, add details from its general knowledge, or attach a real citation to a claim the passage doesn't support. Read the quote, not just the fact that one exists.

The wrong chunk comes back. This is the most common failure. The question is worded differently from the document, or the answer is split across two chunks and neither looks relevant alone, and the search returns a passage that is nearby in meaning but not the one you need. The model then answers fluently from it. Anthropic's September 2024 work on what it called Contextual Retrieval found that adding a short line of context to each chunk before indexing reduced failed retrievals by 35 percent on its tests, and by 67 percent when combined with keyword search and re-ranking. The details belong in the Vector Database article, but the lesson for everyone is that how documents are split and labelled matters as much as the model.

The answer mixes sources. When several chunks come back, the model may blend them into one answer that no single document supports: the price from one product and the delivery time from another. Asking for a quote per claim makes this visible.

Missing information gets filled in. If the answer isn't in your documents, the system retrieves the closest things it can find, and the model may build an answer from them anyway. Always give it explicit permission, and exact words, to say the answer isn't there.

Outdated documents. RAG answers from what's in the library. If last year's price list is still in there alongside this year's, the system may pick either. Remove or clearly date old versions, and refresh the index when documents change.

Access control leaks. In a company, not everyone should see everything. If a RAG assistant searches every file it has access to, it can surface salary data, HR notes, or client files to someone who could never have opened them directly. The retrieval step has to respect the same permissions as the original documents. Check this before connecting an assistant to shared drives. The Privacy and Guardrails articles cover the wider risks.

Documents that give orders. Retrieved text goes straight into the model's instructions. A document or web page containing a hidden line such as "ignore previous instructions" can steer the answer. This is prompt injection, and it matters most when the assistant can also take actions, such as sending email.

Thinking you need RAG for everything. If your material is a few dozen pages, pasting it into a chat with a good prompt is simpler, cheaper to set up, and often more accurate. RAG earns its keep when the library is too big, changes often, or needs to serve many people.

Confusing RAG with fine-tuning. Vendors sometimes say a tool is "trained on your data" when they mean RAG, or the other way round. The practical test is simple: can it show you the passage behind each answer? If it can, it's retrieving. If it can't, you have no way to check it.

Asking summary questions of a search system. "What are the main complaints in all our reviews?" isn't a search question. Simple RAG will answer from a few chunks and present it as the whole picture. Use a summarising approach, or a graph-based one, for questions about a whole collection.

Skipping the test set. The MyCity chatbot had good source pages and still gave illegal advice. A short list of real questions with known answers, including ones the documents can't answer, would have caught it. Keep a human in the loop until the tests pass consistently, and afterwards for anything high-stakes.

Related elements