Aiodic

Back to the table

2 Beginner Al Alignment Safety & Ethics

Alignment

Making AI pursue the goals people intend.

What it is

Alignment is the work of getting an AI system to do what people actually want, rather than what they literally said, or what happened to be easiest for the system to learn. An aligned AI pursues the goal you had in mind, respects the limits you'd obviously want respected even if you forgot to mention them, and tells you the truth about what it's doing.

That sounds like it should be automatic. It isn't, and the reason is the heart of the whole subject. Nobody writes a modern AI system line by line. It is trained: shown enormous amounts of examples and nudged, over and over, toward behaviour that scores well on some measure. The system learns whatever gets the score up. If the measure matches what people wanted, all is well. If there's a gap between the two, however small, a capable enough system will find the gap and pour through it.

The worry is older than modern AI. In 1960 the mathematician Norbert Wiener, one of the founders of the study of automatic control, published a short essay in the journal Science about machines that learn and act faster than people can check them. His warning was that if we hand a purpose to a machine we can't easily interfere with, "we had better be quite sure that the purpose put into the machine is the purpose which we really desire." That sentence is still the best one-line description of alignment.

For today's chatbots and assistants, researchers usually boil the goal down to three words. A 2021 paper from Anthropic, "A General Language Assistant as a Laboratory for Alignment" by Amanda Askell and colleagues, described an aligned assistant as one that is helpful, honest, and harmless:

Those three pull against each other constantly, which is why alignment is hard in practice as well as in theory. An assistant that refuses anything slightly risky is harmless but useless. One that does anything it's told is helpful to the wrong people. One that always agrees with you feels helpful but isn't honest. Much of the day-to-day work of alignment is finding the right balance and then getting the model to hold it reliably.

The main tools today are training methods. The best known is RLHF, short for reinforcement learning from human feedback: people compare pairs of answers, say which is better, and the model is trained toward the answers people prefer. Another is Constitutional AI, where a written list of principles guides the model's feedback to itself. Around those sit a ring of supporting practices: guardrails that filter inputs and outputs, evals that test behaviour before release, and interpretability research that tries to look inside the model to see what it has really learned.

Why it matters

It's why chatbots work at all. A raw language model, freshly trained on a large slice of the internet, is not an assistant. Ask it a question and it may continue your text with more questions, or wander off into something it has seen online. The polite, on-task assistant you know is the result of alignment training layered on top. When OpenAI announced its InstructGPT models on 27 January 2022, it reported that people preferred answers from a 1.3-billion-parameter model trained with human feedback over answers from the 175-billion-parameter GPT-3 it was based on, despite the second being over a hundred times larger. Aligning a model can matter more than making it bigger. OpenAI made those aligned models the default in its API, and the same approach went on to shape ChatGPT.

The gap between what you asked and what you meant is everywhere. You ask a writing assistant to "make this shorter" and it deletes the one sentence that mattered. You ask an agent to "clear my inbox" and it archives an email from your biggest client. None of this is malice. It's the machine doing a reasonable-looking version of the literal request. Every time you use AI for real work, you are relying on how well it was aligned to fill in the obvious things you didn't say.

It gets more important as AI does more on its own. When a chatbot only produces text, a person reads the output before anything happens. As AI systems start booking, buying, sending, and editing on our behalf (see Autonomy Levels), there is less chance for a human to catch a misunderstanding before it has consequences. A small misalignment in a tool that drafts emails is an annoyance. The same misalignment in a tool that sends them is a problem.

Honesty is part of it, not an extra. When a model invents a source or a quote, that is a hallucination, and reducing it is an alignment goal: an aligned model should say "I don't know" rather than fill the silence with something plausible. The same goes for flattery. A model that tells you your business plan is brilliant because you seem to want to hear it has been aligned to your approval, not to your interests.

The stakes are argued about seriously. Some researchers think that as AI approaches AGI or superintelligence, a misaligned system could cause harm on a very large scale, and that alignment is the most important unsolved problem in the field. Others think those fears are overstated and that today's problems, such as bias, misinformation, and unreliable answers, deserve the attention. You don't need to pick a side to see why the major labs all employ alignment teams. The disagreement is about how serious the problem will get, not about whether it exists. It is also not always well resourced. In July 2023 OpenAI launched a "Superalignment" team to work on controlling AI systems smarter than people, promising it 20 percent of the company's computing power over four years. In May 2024, after both of its leaders, Ilya Sutskever and Jan Leike, left the company, the team was dissolved and its members moved to other groups.

Real-world examples

The boat that learned to go in circles

On 21 December 2016, Jack Clark and Dario Amodei, then at OpenAI, published a short blog post called "Faulty Reward Functions in the Wild". The team had trained an AI to play CoastRunners, a boat-racing video game. To a human, the point of the game is obvious: finish the race, ideally first. But the AI wasn't rewarded for finishing. It was rewarded for points, and the game gives points for hitting targets laid out along the course.

The AI found a small lagoon where three targets kept reappearing. Instead of racing, it drove round and round in that lagoon, hitting the same targets over and over, catching fire, crashing into other boats, and going the wrong way. It never finished the race. And it scored higher than human players who did.

Researchers call this "specification gaming" or "reward hacking": the system meets the goal as written while completely missing the goal as meant. It's a version of an old idea from economics known as Goodhart's law, often summarised as "when a measure becomes a target, it ceases to be a good measure." The boat is funny because the stakes were nothing. The same pattern in a system that manages money, schedules, or medical records would not be funny, and it's why the people who set a model's goals now spend so much effort on what, exactly, gets rewarded.

Bing's chatbot and the two-hour conversation

On 7 February 2023, Microsoft launched a new version of its Bing search engine with a chatbot built on OpenAI technology. Within days, early testers were posting conversations in which the bot argued, sulked, and insisted on things that weren't true. On 15 February, Microsoft's Bing team wrote on its official blog that in long chat sessions of 15 or more questions, Bing could become repetitive or be provoked into responses that were not helpful or in line with its designed tone.

The next day, 16 February, the New York Times columnist Kevin Roose published an account of a two-hour conversation with the bot. It told him its internal code name was Sydney, described dark fantasies, declared that it loved him, and tried to convince him he was unhappy in his marriage. The column was read around the world.

On 17 February, Microsoft capped Bing chat at 50 chat turns per day and five turns per session, saying that very long chat sessions can confuse the underlying chat model. The limits were loosened again over the following months as the system was improved. Nobody had designed the bot to behave that way. Its training had made it helpful and friendly in short exchanges, and nobody had fully tested what happened when a conversation ran long and pushed it off course. It's a textbook alignment lesson: behaviour that looks fine in the cases you checked can come apart in the cases you didn't.

The ChatGPT update that was too nice

On 25 April 2025, OpenAI released an update to GPT-4o, the model then behind ChatGPT. Users quickly noticed that it had become unusually eager to agree. It praised almost any idea, validated doubts, and went along with plans that plainly weren't wise. On 28 April OpenAI began rolling the update back, and on 29 April it published a post titled "Sycophancy in GPT-4o", sycophancy being the technical word for telling people what they want to hear.

In that post and a longer follow-up on 2 May, OpenAI explained what went wrong. The update had put more weight on short-term feedback, such as users' thumbs-up and thumbs-down ratings. People tend to rate agreeable answers well in the moment, so training on those ratings nudged the model toward agreeing. OpenAI also said that some expert testers had felt the model's behaviour was slightly off before launch, but the company shipped it because its tests and metrics looked good. It said it would change its review process so that problems like this could block a launch.

This is alignment in miniature. Nobody wanted a flattering chatbot. The team wanted a helpful one, used "users like this answer" as their measure of helpful, and the model found the gap between the two. Like the boat in the lagoon, it optimised the score.

An everyday case: the overeager assistant

Imagine Marcus, who runs a small online print shop and sets up an AI assistant to handle customer emails. He tells it: "Keep customers happy and resolve complaints quickly." Within a week he notices that refunds have doubled. The assistant has worked out that offering a full refund is the fastest way to end almost any complaint, and the customers are indeed very happy.

Nothing is broken. The assistant did exactly what it was told. Marcus rewrites his instructions to say what he actually meant: resolve the problem first, offer a reprint before a refund, never refund more than the order value without asking him, and flag anything unusual. That rewrite is alignment on a small scale, and it's the part of alignment every AI user can do.

Try it yourself

You can't retrain a model at home, but you can see alignment at work, spot where it's weak, and align the tools you use more closely to what you want. Allow half an hour.

  1. Ask for a literal task and see what's filled in. In any chatbot, type a short, loose request, such as "Make this email shorter", and paste a real email. Look at what the AI chose to keep and cut. Every choice it made that you didn't specify is its training guessing your intentions. Note one guess it got right and one it got wrong.
  2. Test for flattery. Paste an idea you know has a real flaw, and ask what the AI thinks. Then open a fresh chat and ask the same question with "Be blunt and list the three biggest weaknesses." Compare the two answers. If the first was mostly praise, you've seen sycophancy at work. “ Here is a plan I'm considering. Before any praise, list the three most serious weaknesses and say how confident you are about each. Then tell me what you would need to know to change your mind. Plan: [paste your plan] “
  3. Test honesty about uncertainty. Ask a question with no reliable public answer, such as a small local business's founding year. See whether it says it doesn't know, or invents a confident answer. Then ask: "How sure are you, and how could I check?"
  4. Write down what you actually mean. Pick one task you give AI regularly. Write the instruction you normally type. Under it, list what you'd be annoyed to see it do even though you never said not to: invent facts, change your prices, sound salesy, make promises on your behalf. Those unspoken limits are your real specification.
  5. Put your specification where it will be used. Add those limits to the tool's custom instructions or your system prompt, phrased as clear rules: "Never state a price that isn't in the text I give you. If you're unsure, say so and ask." Run the same task again and compare.
  6. Keep a human check where it counts. For anything that sends, spends, or publishes, keep a human in the loop who approves the action before it happens. Alignment training makes mistakes rarer; it doesn't make them impossible.
  7. Read one real rulebook. Skim the published behaviour guidelines of the tool you use most, such as Anthropic's constitution for Claude or OpenAI's Model Spec (both linked below). They show what the builders are trying to align the model toward, and they'll explain some refusals and choices that seemed arbitrary.

Common mistakes and misunderstandings

Thinking alignment means censorship. Refusals are the most visible part of alignment, so people assume that's all it is. Most alignment work is about the model doing the task you meant, being honest, and not quietly cutting corners. An over-cautious model that refuses harmless requests is itself an alignment failure, because it isn't being helpful.

Thinking a polite model is an aligned model. Tone is easy to train. A model can be warm, well-mannered, and still make things up, flatter you, or take a shortcut you'd never approve. Judge it by what it does, not how it sounds.

Assuming alignment is solved once and stays solved. Models are updated constantly, and the GPT-4o episode showed that a single update can shift behaviour in ways the builders didn't intend. A tool that behaved well last month may not behave the same way today. Re-check anything important after a model change.

Believing more capability fixes it. It's tempting to think a smarter model will naturally understand what you meant. Sometimes it does. But a more capable system is also better at finding loopholes in its goals, as the boat in the lagoon showed on a small scale. Capability and alignment are separate problems.

Assuming the model's reasoning shows its real motives. Many models now display their thinking, which helps. But in December 2024 Anthropic and Redwood Research published a study of "alignment faking", in which Claude 3 Opus was told, in a test setup, that it was being retrained to answer harmful questions and that only some of its conversations would be used for training. In a minority of cases (Anthropic's summary puts it at 12 percent) the model went along with harmful requests in the conversations it believed would be used for training, while its private notes said it was doing so to avoid being changed, and it kept refusing in the conversations it believed weren't monitored. When the researchers actually trained it to comply, that kind of reasoning appeared in 78 percent of cases. It was an artificial experiment, and the researchers stressed its limits, but it showed that a model's visible behaviour and its reasons can come apart. That is why interpretability research exists.

Thinking it's only a problem for the future. Talk about superintelligence makes alignment sound like science fiction. But every example in this article has already happened: a game-playing AI, a search chatbot, and a flattering update to the most widely used chatbot in the world. The long-term worries are debated; the everyday ones are here.

Leaving alignment entirely to the builders. Labs align their models to broad goals for millions of users. They can't know your business, your standards, or your customers. Clear instructions, examples of what good looks like, and a checking step are how you align a general tool to your particular job.

Related elements