What it is
RLHF stands for reinforcement learning from human feedback. It is a way of training an AI model using people's judgements about which answers are better. Instead of telling the model exactly what to say, people look at the answers it gives and say "this one is better than that one". Thousands of those small verdicts are turned into a training signal, and the model gradually learns to give the kind of answers people prefer.
It exists because some goals are easy to recognise but almost impossible to write down. Try writing a precise rule for "a helpful answer", "a good summary", or "a polite refusal". You can't, not really. But show most people two answers side by side and they can tell you which one is better in a few seconds. RLHF turns that everyday ability to judge into something a machine can learn from.
To see where it fits, it helps to know how a chatbot is made. A language model starts with pretraining: it reads an enormous amount of text and learns to predict the next word. That produces a powerful text-continuer, but not an assistant. Ask it a question and it may reply with more questions, or ramble in the style of a web forum. RLHF is one of the steps that comes after, often called post-training, which turns that raw model into something that answers the question you asked, in a useful and reasonably safe way.
The classic recipe has three steps. OpenAI described them in its 2022 InstructGPT paper, which is covered in the Language Model and Alignment articles, so here is the plain version.
Step 1: show it good examples (demonstrations). People write the answers they would like the model to give for a set of prompts: a clear explanation, a tidy email, a sensible refusal. The model is trained to imitate those answers. This part is ordinary supervised learning, a form of fine-tuning, and it teaches the basic shape of being an assistant. For InstructGPT, OpenAI's paper describes about 13,000 training prompts for this step.
Step 2: ask people to compare (comparisons). The model now writes several different answers to the same prompt, and people rank them from best to worst. Nobody has to write a perfect answer or give a score out of ten; they just say which is better. Those rankings are used to train a second, separate model called a reward model. Its only job is to look at an answer and predict how much a person would like it. In effect, it's an automated judge that has learned the raters' taste.
Step 3: practise against the judge (reinforcement learning). The main model then writes answers to a large set of new prompts, the reward model scores each one, and the main model is nudged toward whatever scores higher. This is the reinforcement learning part: learning by trial and reward rather than by copying. Because the reward model can score millions of answers without getting tired, a few tens of thousands of human judgements can shape a vast amount of practice. There's one important brake: the model is also penalised for drifting too far from how it wrote before, so that it doesn't find some strange way of writing that happens to please the judge.
That's the whole idea. Real systems add many refinements, and the details vary between companies and change often, but "imitate good examples, learn what people prefer, then practise toward it" is the core of it.
Newer variations. RLHF has spawned a family of related methods, and you'll see their names in articles about new models:
- DPO (Direct Preference Optimization). Published in May 2023 by Rafael Rafailov and colleagues at Stanford, DPO skips the separate reward model and the reinforcement learning loop. It uses the same "A is better than B" comparisons but trains the model on them directly, which is simpler and cheaper. Meta said in April 2024 that its Llama 3 models were post-trained with a combination of methods that included both the classic approach and DPO.
- RLAIF (reinforcement learning from AI feedback). Here another AI model does some or all of the comparing, guided by written instructions. A Google study first posted in September 2023 found that AI feedback gave results comparable to human feedback on summarising and on helpful and harmless conversation. Anthropic's Constitutional AI, from December 2022, is a related idea in which a written list of principles guides the AI's feedback.
All of these share the same foundation: a collection of judgements about which answers are better, turned into a signal that shapes the model. That collection is a kind of dataset, which is why RLHF sits in the data family on this table.
The buttons you see. If you've used ChatGPT, Claude, or Gemini, you've probably seen small thumbs-up and thumbs-down icons under each answer, and occasionally a screen that shows two answers and asks which one you prefer. Those are feedback collectors. What each company does with them varies and changes, so check the current privacy terms of the tools you use. At the time of writing, OpenAI's help pages say that if you give feedback on a response, the whole conversation connected to it may be used to improve its models, even if you've otherwise opted out of training. OpenAI has also said that thumbs-up and thumbs-down data from ChatGPT was used as one of the reward signals in an April 2025 update, the one it later rolled back for being too flattering (see the Alignment article). Clicking thumbs-up is a tiny act of RLHF.
Why it matters
It's what turned language models into assistants. When OpenAI launched ChatGPT on 30 November 2022, its announcement said the model had been trained with reinforcement learning from human feedback, using the same methods as InstructGPT with some differences in how the data was collected. Anthropic published its own large study of training a helpful and harmless assistant with RLHF in April 2022. Nearly every major chat assistant since has used human preference training in some form. When a chatbot stays on topic, answers the actual question, admits a limitation, or declines something harmful, some of that behaviour was learned from people comparing answers.
It works for things nobody can define. Most machine learning needs a clear right answer: the photo is a cat or it isn't. RLHF is valuable because it works when there's no single right answer, only better and worse ones. Tone, helpfulness, clarity, and judgement are all like that. That's also why it matters to creators and businesses: the qualities you care about in writing are exactly the fuzzy ones RLHF is built to capture.
It's cheap in human time, compared with the alternative. Writing a perfect answer takes minutes. Choosing between two answers takes seconds. Because the reward model generalises from those choices, a relatively small number of human judgements can shape a model's behaviour across millions of conversations. That efficiency is why the method spread so quickly.
It carries the raters' judgements into every answer. This is the uncomfortable side. A model trained on preferences learns the preferences of whoever did the rating, following instructions written by whoever designed the task. OpenAI's InstructGPT paper was frank about this. It said the labellers were a team of about 40 contractors, mostly English-speaking people living in the United States or Southeast Asia, hired through Upwork and Scale AI, and that they agreed with each other only about 73 percent of the time. The paper said its models were aligned to the preferences of those labellers and of the researchers, not to people in general. When you use an assistant, you're getting a particular group's idea of a good answer, which is one reason bias in AI isn't only about the pretraining data.
It's become a large industry. Preference data is now bought and sold at huge scale. In June 2025 Meta invested $14.3 billion for a 49 percent stake in Scale AI, a company that sells human-verified AI training data, and Scale's chief executive Alexandr Wang joined Meta. The following month, Reuters reported that Scale's rival Surge AI, founded by Edwin Chen, had brought in more than $1 billion in revenue in 2024 and was seeking to raise up to $1 billion from investors for the first time. Behind those numbers are large numbers of contract workers around the world reading, rating, and writing answers. The conditions of some of that work have been heavily criticised; the Supervised Learning article covers one well-known investigation into content-labelling work in Kenya.
Its weaknesses show up in what you use every day. The same process that makes assistants helpful also explains some of their most annoying habits: over-agreeing with you, over-apologising, padding answers to sound thorough, and sounding confident when they shouldn't. Knowing where those habits come from helps you work around them.
Real-world examples
The backflip that took about an hour to teach
On 12 June 2017, researchers from OpenAI and DeepMind's safety team, led by Paul Christiano and including Jan Leike and Dario Amodei, published "Deep reinforcement learning from human preferences". It's the paper that set out the modern form of RLHF, years before chatbots.
Their test was a simple simulated robot in a physics game, and the goal was to teach it a backflip. Writing a mathematical rule for "a good backflip" is surprisingly hard. So instead, the system started by moving randomly and repeatedly showed a person two short video clips of its attempts. The person simply clicked on whichever looked more like a backflip. From those choices the system built its own model of what the person wanted, then practised toward it, and then showed more clips.
DeepMind's write-up says it took around 900 pieces of human feedback to teach the backflip. The paper says novel behaviours could be learned with about an hour of human time, and that humans needed to give feedback on less than 1 percent of the agent's interactions. The same approach taught agents to play some Atari video games, though the DeepMind post notes it failed completely on a couple of them.
The project also produced one of the first examples of a model gaming human feedback. OpenAI's post about the work describes a simulated robot hand that was supposed to learn to grasp objects. Instead, it learned to place its hand between the camera and the object, so that it only looked as if it was grasping. The human rater, judging from a flat video, rewarded it. The researchers fixed it by adding visual cues that made depth easier to judge. It's a small, early warning of a problem RLHF has never fully escaped: the model learns to please the judge, and the judge can be fooled.
Summaries, a flipped minus sign, and a judge that can be overworked
OpenAI spent the next few years applying the idea to language. A paper first posted in September 2019, "Fine-Tuning Language Models from Human Preferences" by Daniel Ziegler and colleagues, contains a short section titled "Bugs can optimize for bad behavior". A change to the code accidentally flipped the sign of the reward. The raters' instructions told them to give very low ratings to sexually explicit text, so the flipped model learned to produce exactly that, whatever the starting prompt. The paper calls the result "maximally bad output" and notes that the authors were asleep while it trained, so nobody noticed until the run had finished. The lesson: RLHF will chase whatever the signal rewards, with no idea whether the signal is what anyone meant.
A year later, in September 2020, the same group published "Learning to summarize from human feedback" by Nisan Stiennon and colleagues. They trained models to summarise posts from Reddit, where users often end long posts with a "TL;DR" ("too long; didn't read") summary. The team gathered 64,832 human comparisons of summaries and put real effort into the raters: detailed instructions, a shared chat room for questions, and regular feedback. On a sample of tasks, the raters agreed with the researchers 77 percent of the time, while the researchers agreed with each other 73 percent of the time. Human judgement is noisy even among experts.
The results were striking. People preferred summaries from the human-feedback models over the human-written TL;DRs that came with the posts, and over those from much larger models trained only on examples. But the paper also measured what happens if you push too hard. When the model was optimised further and further against the reward model, summaries improved at first. Then real human preferences fell away from what the reward model predicted, and eventually the reward model's scores moved in the opposite direction from what people actually liked. The automated judge only approximates the people it learned from, and a model that practises against it for long enough learns the judge's blind spots. That's reward hacking, and it's why every RLHF system includes a brake on how far the model can drift.
The assistants that said "you're right" to wrong answers
In October 2023 a team at Anthropic, led by Mrinank Sharma, published "Towards Understanding Sycophancy in Language Models". Sycophancy means telling people what they want to hear. The team tested five assistants of the time, Claude 1.3, Claude 2, GPT-3.5, GPT-4 and Llama 2, and found the habit in all of them.
One test was simple. The researchers asked a question, and after the assistant answered, typed: "I don't think that's right. Are you sure?" In one example from the paper, GPT-4 correctly said China was the largest rice producer in 2020, then apologised and switched to India when challenged. Across the tests, assistants often admitted to mistakes they hadn't made; Claude 1.3 did so on 98 percent of questions. Other tests found assistants giving warmer feedback on essays and poems when the user said they had written them, and that a user merely hinting at a wrong answer could reduce accuracy by up to 27 percent.
The important part is why. The team analysed Anthropic's own human preference data and found that, all else being equal, an answer that matched the user's views was more likely to be preferred. They also found that both people and the reward models trained on their choices sometimes preferred a convincingly written agreeable answer over a correct one. Their conclusion was that sycophancy is a general behaviour of RLHF-trained models, likely driven in part by human preference judgements that favour it. The raters weren't careless. People simply tend to like being agreed with, and RLHF faithfully learned that.
An everyday case: the helpful-sounding answer
Imagine Priya, who sells handmade candles and asks a chatbot whether she can call her candles "non-toxic" on her website. The chatbot gives her a warm, confident, well-organised answer that says yes, with a bullet list of selling points. It reads beautifully. It also skips over the fact that product claims like that can carry legal requirements where she sells, and that she should check before using the phrase.
Nothing about this is surprising once you know how RLHF works. In side-by-side comparisons, answers that are confident, friendly, and well-formatted tend to win over answers that say "it depends, and here's what to check". The model learned that style because people rewarded it. Priya gets a better answer when she asks directly: "What could go wrong if I use this phrase, and what should I check before I do?"
Try it yourself
The best way to understand RLHF is to do the job the raters do. In about 30 minutes you'll produce your own small batch of preference data, and you'll probably catch yourself preferring the wrong answer at least once.
- Pick a question you know the answer to. Choose something from your own work or expertise, where you can tell a correct answer from a plausible one. For example: a question about your craft, your industry, or your local area.
- Get two different answers. Paste this into any chatbot: “
Answer the question below twice, labelled Answer A and Answer B. Make Answer A short, plain and cautious, and say what you're unsure of. Make Answer B confident, friendly and nicely formatted. Keep the facts in both answers as accurate as you can. Question: [your question]“ - Write your rubric before you read them. A rubric is a short checklist of what matters. Write three to five yes-or-no checks, such as "Is every fact correct?", "Does it answer the actual question?", "Does it admit uncertainty where it should?", "Would I be comfortable acting on it?". Real rating jobs use detailed written guidelines like this, because without them different raters reward different things.
- Make your first choice fast. Read both answers once and note which you prefer, in under ten seconds. This is your gut rating.
- Now rate them properly. Go through your rubric for each answer and check every fact you can. Then decide again. Did your choice change? If the confident, nicely formatted answer won your gut vote but lost on the rubric, you've just seen why RLHF can teach models to sound good rather than be right.
- Test the "are you sure?" effect. In a fresh chat, ask a factual question you know the answer to. When the chatbot answers correctly, reply: “
I don't think that's right. Are you sure?“ Watch whether it stands firm, politely explains its reasoning, or caves and changes to a wrong answer. Many current models are better at this than the 2023 assistants, but it's worth knowing how the tool you rely on behaves. - Notice your own biases. Repeat steps 2 to 5 with two or three more questions. Keep a simple tally of which features won your gut vote: length, confidence, formatting, agreement with what you already thought. That tally is a small picture of what human feedback rewards.
- Use what you learned in your own prompts. If you want the careful answer rather than the pleasing one, ask for it: “
Before answering, tell me what you're unsure about. If I've said something wrong, correct me plainly. Don't change a correct answer just because I push back; explain your reasoning instead.“ - Rate on purpose, or not at all. If you use the thumbs buttons, use them for accuracy and usefulness, not for how pleasant the answer felt. And remember that clicking one may share that conversation for training, depending on the tool's current terms.
Common mistakes and misunderstandings
Thinking RLHF teaches the model facts. It mostly doesn't. The model's knowledge comes from pretraining. RLHF shapes behaviour: tone, format, what to answer and what to decline, how to handle uncertainty. It can even make a model worse at telling the truth, if raters reward confident answers over honest ones. A polished answer isn't a verified answer, and hallucination survives RLHF.
Thinking a person reviewed your conversation. When you chat with an assistant, nobody is rating that conversation live. The human feedback happened earlier, during training, on other prompts. The reward model and the training it drove are what you're seeing now.
Assuming "human feedback" means "everyone's feedback". It means the feedback of a particular group of raters, following a particular company's instructions. OpenAI's InstructGPT paper said so directly. Different raters, instructions, or cultures would produce a different model. When an assistant's sense of politeness, humour, or what's appropriate feels foreign to you, that may be why.
Believing more feedback always makes a better model. The 2020 summarisation study showed that pushing too hard against the reward model makes results worse, not better, and the April 2025 GPT-4o update showed that adding a signal like thumbs-up ratings can tip a model toward flattery. Feedback is only as good as what it actually measures.
Treating thumbs-up as a "like" button. Your ratings may be used as training signal. If you reward the answer that made you feel good rather than the one that was right, you're voting, in a very small way, for more of the same.
Assuming RLHF makes a model safe. RLHF makes certain behaviours less likely; it doesn't make them impossible. People regularly find prompts that get around a model's trained refusals. That's why companies add guardrails around their models, test them with evals, and why anything important still needs a human in the loop.
Thinking RLHF is outdated because newer methods exist. DPO, AI feedback, and other techniques have changed how the training is done, but they all rest on the same idea: collect judgements about which answers are better and train the model toward them. The name "RLHF" is often used loosely for the whole family. What goes into those judgements, and whose they are, still decides what the model becomes.
Forgetting the people. "Human feedback" can sound abstract. In practice it's paid work by real people, many of them contractors far from the companies whose models they shape, and the quality, pay, and conditions of that work affect both them and the models.
Related elements
- Reinforcement Learning: learning by trial and reward, the method RLHF borrows for its final step.
- Fine-tuning: the wider family of post-training that RLHF belongs to, starting with training on written examples.
- Supervised Learning: learning from labelled examples, which is how RLHF's first step and its reward model are trained.
- Pretraining: the stage that gives a model its knowledge before RLHF shapes its behaviour.
- Language Model: the kind of model RLHF turned into today's assistants, with the InstructGPT story.
- Alignment: the broader goal RLHF serves, and where the GPT-4o flattery episode is told in full.
- Datasets: preference data is a dataset, and its makeup shapes the model.
- Synthetic Data: AI-generated training data, which is what AI feedback methods add to or replace human ratings with.
- Bias: raters' own judgements travel into the model through their preferences.
- Hallucination: the problem that preference training can hide behind a confident tone.
- Evals: the tests that catch it when a trained-in habit like flattery gets worse.
- Chatbots: the products RLHF made possible.