Aiodic

Back to the table

37 Intermediate Rl Reinforcement Learning Machine Learning

Reinforcement Learning

Learning by trial, error, and reward.

What it is

Reinforcement learning is a way of teaching a computer by letting it try things and telling it how well each attempt went. Nobody shows it the right answer. It acts, it gets a score, and over thousands or millions of attempts it works out which actions tend to lead to high scores.

If you've ever trained a dog, you already understand the idea. You don't explain to a puppy, in words, what "sit" means. You wait for it to lower its back end, you say "good dog" and hand over a treat, and you repeat. The puppy tries all sorts of things (jumping, barking, rolling over), and the behaviour that earns treats slowly becomes the behaviour it chooses. Reinforcement learning works on the same principle, with a computer program in place of the dog and a number in place of the treat.

It's one of the three broad styles of machine learning. In supervised learning, the computer studies examples that come with the correct answer attached. In unsupervised learning, it looks for patterns in data that has no answers at all. Reinforcement learning sits in between: there's no answer key, but there is feedback, and the feedback often arrives late. A chess program doesn't learn whether move 12 was good until the game ends 40 moves later.

The five words you need

Almost every explanation of reinforcement learning uses the same five terms. Here they are, using a video game as the running example: imagine a character who has to cross a level full of pits to reach a flag.

The whole learning process is a loop. The agent looks at its situation (its "state"), picks an action according to its policy, receives a reward and a new situation, and nudges its policy so that actions which led to good rewards become more likely next time. Then it goes round again, and again, often millions of times.

Exploration versus exploitation

There's a tension at the heart of every reinforcement learning system, and it's the same one you face when choosing a restaurant. You can go back to the place you know is good (exploitation: using what you've learned), or you can try somewhere new that might be better or might be awful (exploration: gathering new information).

An agent that only exploits gets stuck. Suppose the game character discovers early on that standing still on the first platform avoids the −100 for falling. If it never experiments, it will stand there forever and never find the flag. An agent that only explores never settles down to use what it has learned. It keeps jumping into pits to "see what happens".

Good systems balance the two. A common trick is to act on the best-known choice most of the time and take a random action a small fraction of the time, say one move in ten, with that fraction shrinking as the agent gets more confident. You'll see this setting, often called epsilon, in the browser demo in "Try it yourself" below.

Where the idea came from

The idea of learning from reward and punishment is older than computers. Animal psychologists studied it in the early twentieth century, and Alan Turing suggested in 1950 that a machine might learn through rewards and punishments. The modern field was built in the 1980s by Andrew Barto and his PhD student Richard Sutton at the University of Massachusetts Amherst, who turned the idea into precise mathematics and working algorithms. Their textbook, Reinforcement Learning: An Introduction, first appeared in 1998 and is still the field's standard reference.

Why it matters

It can learn things nobody knows how to teach. Supervised learning needs someone to supply the right answers, which means it can at best match the people who wrote them. Reinforcement learning only needs a way to score results. That's why it produced the moments where AI did something its makers hadn't thought of, such as the Go move described below. If you can measure success, the system can in principle find routes to it that no human would have suggested.

It's how AI learns to act, not just to predict. Most machine learning answers questions: is this email spam, what's in this photo. Reinforcement learning is about sequences of decisions, where each choice changes what happens next. That makes it the natural fit for games, robots, and any system that has to take a series of steps towards a goal, including the AI agents now being built to carry out tasks on a computer.

It shaped the chatbots you use. A raw language model trained only to predict the next word is knowledgeable but not especially helpful. Reinforcement learning is one of the main tools used to turn it into an assistant. The best-known method is RLHF, reinforcement learning from human feedback, in which people rate the model's answers and those ratings become the reward. When the Association for Computing Machinery (ACM) announced the 2024 Turing Award for Barto and Sutton, it named ChatGPT's use of RLHF as one of the field's major applications.

It's behind the newest "thinking" models. Since 2024, AI companies have used reinforcement learning to train models that work through a problem step by step before answering, rewarding them when the final answer is correct. That's covered briefly in the examples below and in more depth under Chain of Thought and Extended Thinking.

It fails in instructive ways. An agent will chase whatever the reward actually measures, not what you meant. The famous case is a boat-racing agent that learned to circle forever collecting points instead of finishing the race, described on the Alignment page. Understanding reinforcement learning makes it much easier to understand why AI systems sometimes behave in ways that are technically "correct" and obviously wrong.

Real-world examples

DeepMind's Atari player learns 49 games from the screen (February 2015)

On 26 February 2015, the London company DeepMind (bought by Google the year before) published a paper in the journal Nature titled "Human-level control through deep reinforcement learning". It described a program the team called a deep Q-network, or DQN, that learned to play classic Atari 2600 video games.

What made it remarkable was how little it was told. The agent saw only the raw pixels on the screen and the game score, and it could press the same buttons a human player could. It wasn't told the rules of any game or what the objects on screen meant. The same program, with the same settings, was trained separately on 49 different games. According to the paper and Google's announcement, it beat the best earlier machine learning methods on 43 of the 49, and on more than half it reached over 75 percent of the score of a professional human games tester.

The most-quoted detail came from Breakout, the game where you bounce a ball off a paddle to knock out a wall of bricks. Google's write-up reports that the agent learned to dig a tunnel through one end of the wall so the ball could get behind the bricks and knock them out from above, a strategy experienced human players use. Nobody told it to do that. The score went up faster when it did, so it did it more.

AlphaGo and move 37 (March 2016)

Go is an ancient board game with far more possible positions than chess, and for decades it was considered a game computers couldn't master. In March 2016, DeepMind's program AlphaGo played a five-game match in Seoul against Lee Sedol, one of the strongest players in the world, and won four games to one. DeepMind says the match was watched by more than 200 million people.

AlphaGo was trained in two stages. First it studied thousands of games played by human experts, which is supervised learning. Then it played against versions of itself over and over, learning from wins and losses, which is reinforcement learning. The second stage is what let it go beyond copying people.

The moment everyone remembers is move 37 of game two. AlphaGo placed a stone in a spot that professional commentators first took to be a mistake. By DeepMind's own estimate, a human player would have chosen that move with a probability of about 1 in 10,000. It turned out to be strong, and AlphaGo won the game. Lee Sedol had his own answer two games later: his move 78 in game four, which DeepMind itself describes as just as unlikely and inventive, won him the only game a human took in the match.

AlphaZero teaches itself three games from the rules alone (2017 to 2018)

AlphaGo had started from human games. Its successor did not. In December 2017, DeepMind posted a preprint describing AlphaZero, and the full peer-reviewed paper appeared in the journal Science in December 2018. AlphaZero was given nothing but the rules of chess, shogi (Japanese chess), and Go. It began with random play and improved purely by playing millions of games against itself, adjusting its policy after each win, loss, or draw.

According to DeepMind, it took about 9 hours of training to reach superhuman strength at chess, 12 hours at shogi, and 13 days at Go. It then beat the leading specialist programs in each game, including the top chess engine Stockfish. Strong human players who studied its games commented on its unusually dynamic, attacking style, and the former world champion Garry Kasparov was among those who said people could learn from what it discovered. It's the clearest demonstration of reinforcement learning's big promise: given a clear reward (win the game) and enough practice, a system can discover strategies instead of inheriting them.

A robot hand learns to handle a Rubik's Cube (October 2019)

Games are tidy. The physical world is not. On 15 October 2019, OpenAI announced that a human-like robot hand had learned to solve a Rubik's Cube one-handed. The hand's control system was trained entirely in computer simulation, using reinforcement learning, and then transferred to the real robot.

The hard part was that no simulation is perfectly realistic. OpenAI's solution, which it called automatic domain randomization, kept making the simulated world harder and stranger (changing physical details such as friction and the size of the cube) as the agent improved, so the policy learned to cope with conditions it hadn't seen. To test it, the team disturbed the real hand with a rubber glove, a blanket, and a stuffed toy giraffe.

It's also a good lesson in reading headlines carefully. The reinforcement learning handled the finger movements, the physical manipulation. The order of turns needed to solve the puzzle came from a standard cube-solving algorithm, not from the AI working it out. And the hand wasn't perfect: it succeeded about 60 percent of the time, and only about 20 percent of the time on the hardest scrambles.

The founders win the Turing Award (March 2025)

On 5 March 2025, the ACM announced that Andrew Barto and Richard Sutton had won the 2024 A.M. Turing Award, often described as the Nobel Prize of computing, "for developing the conceptual and algorithmic foundations of reinforcement learning". The award carries a $1 million prize, funded by Google. The ACM's announcement traced the work back to the early 1980s, when Barto and Sutton began, in its words, to formulate reinforcement learning as a general problem framework, motivated by observations from psychology. It named AlphaGo, robot learning, and ChatGPT's training as examples of where their ideas ended up.

DeepSeek-R1-Zero learns to reason from rewards alone (January 2025)

In January 2025, the Chinese company DeepSeek published a paper on its reasoning models (the peer-reviewed version appeared in Nature on 17 September 2025). One of them, DeepSeek-R1-Zero, was trained on maths, coding, and logic problems with reinforcement learning only, without first being shown human-written examples of good reasoning. The reward was simple and checked by rules: points for a correct final answer, and points for putting the reasoning and the answer in the required format.

The paper reports that R1-Zero's average score on the 2024 AIME, a hard American maths competition, rose from 15.6 percent to 77.9 percent during training. More interesting was how: the model started writing longer reasoning, checking its own work, and going back to reconsider, including what the authors called an "aha moment", a sudden increase in the word "wait" as it paused to re-examine a step. It also had flaws, including poor readability and a habit of mixing English and Chinese in the same answer, which is why DeepSeek's released model, R1, added other training stages. The Inference page covers R1's launch and what it did to AI prices; the point here is that a reward for right answers was enough to produce behaviour that looks a lot like careful thinking.

An everyday scenario: the coffee shop's discount emails

This one is illustrative, not a real case. Imagine a small coffee shop that sends a weekly email offer. It could offer a free pastry, 10 percent off, or double loyalty points. Each week, the owner sends one offer and counts how many people come in and use it. That count is the reward.

If the owner picks whichever offer did best the first week and never changes, that's pure exploitation, and they may never find out that double points would have worked better. If they rotate at random forever, that's pure exploration, and they keep wasting weeks on the weakest offer. A sensible owner mostly sends the current best and occasionally tries another, just in case. That's reinforcement learning's explore-or-exploit balance done by hand, and some email and advertising tools automate exactly this kind of testing.

Try it yourself

You can watch reinforcement learning happen in a web browser in five minutes, and you can play the explore-or-exploit game at the kitchen table with no screen at all. No account or coding needed for either.

Part one: watch an agent learn a maze

  1. Open the REINFORCEjs GridWorld demo made by the AI researcher Andrej Karpathy: cs.stanford.edu/people/karpathy/reinforcejs/gridworld_td.html. It works on a desktop or laptop browser.
  2. Look at the grid. The agent starts in the top-left corner. Grey squares are walls. One square in the middle gives a reward of +1 and some squares give −1. The agent can move up, down, left, or right.
  3. Watch for a minute. The agent starts out wandering almost at random. Little arrows appear on each square: that's the policy, the agent's current best guess of which way to go from there. The colours show how valuable the agent thinks each square is.
  4. Click "Go fast". Within a short while, the arrows line up into a clear route to the +1 square that avoids the −1 squares. Nobody drew that route. It came from trial, error, and reward.
  5. Now change the world. Click any square to select it, then drag the "Cell reward" slider to make it strongly negative, or turn a −1 square into a reward. Watch the agent struggle, then reroute its policy.
  6. Find the "Exploration epsilon" slider. Set it to zero and reset the agent with "Reinit agent": with no exploration it can get stuck on a poor route. Set it high and it keeps wandering even after it knows the way. Try something in between.

Part two: the coin-and-cup game (offline)

This is a classic reinforcement learning puzzle called a "multi-armed bandit", named after slot machines. You need a friend, three cups, and some coins.

  1. Your friend secretly decides a hidden payout rate for each cup, for example cup A pays out 2 times in 10, cup B 5 times in 10, cup C 8 times in 10. They write this down and don't tell you.
  2. You get 30 turns. On each turn, you point at a cup. Your friend rolls a ten-sided die (or uses a phone's random number generator) and, if the number is within that cup's rate, drops a coin in front of you.
  3. Keep a tally of pulls and coins for each cup. Your goal is to end with as many coins as possible.
  4. Play once as a pure exploiter: stick with whichever cup pays first. Play again as a pure explorer: rotate cups evenly all game. Play a third time with a balanced plan: try each cup a few times, then mostly pick the best, with an occasional check on the others.
  5. Compare your totals. The balanced plan usually wins, and you'll feel exactly the tension that every reinforcement learning system has to manage.

Part three: ask a chatbot to explain it back

Paste this into any chatbot to check your understanding:

I've just learned about reinforcement learning. Here is my explanation
in my own words: [write two or three sentences].
Please tell me what I got right, what I got wrong, and give me one new
everyday example of an agent, environment, action, reward and policy.
Keep it non-technical.

Common mistakes and misunderstandings

Thinking the AI "wants" the reward. The agent doesn't feel pleasure or desire. The reward is just a number used to adjust its settings. Words like "wants" and "tries" are convenient shorthand, but they can mislead people into imagining motives that aren't there.

Assuming it learns what you meant. It learns what you measured. If the reward counts points, the agent will pursue points by any route that works, including loopholes nobody spotted. This is called reward hacking, and it's one of the central problems in Alignment. Designing a reward that captures what you really want is often harder than running the training.

Believing the headline without the footnotes. Reinforcement learning results often come with important fine print. The Rubik's Cube hand used a separate algorithm to decide the moves and succeeded about 60 percent of the time. AlphaGo started from human games. Read past the first paragraph before concluding what the AI "taught itself".

Thinking it's cheap or quick. Agents usually need enormous amounts of practice: millions of games, or vast amounts of simulated practice. That's why so much of the work happens in games and simulations, where practice is fast and mistakes are free. Learning by trial and error in the real world, where errors cost money or cause harm, is far harder.

Assuming skill transfers. An agent that has mastered one game or one simulated room is usually useless in a different one. The Atari agent was trained separately on each of its 49 games. General-purpose skill remains an open research problem, which is part of what World Models and Embodied AI research is trying to address.

Confusing RL with RLHF. Reinforcement learning is the general method. RLHF is one specific use of it, where the reward comes from human ratings of a language model's answers. Newer reasoning models also use rewards from automatic checks, such as whether a maths answer is correct. All are reinforcement learning, with different sources of reward.

Thinking chatbots learn from you in real time. Reinforcement learning happens during training, before the model is released. When you give a thumbs-up or thumbs-down in a chatbot, that feedback may be collected for future training, depending on the company's policies, but it doesn't change the model you're talking to in that moment.

Related elements