Aiodic

Back to the table

18 Beginner Bi Bias Safety & Ethics

Bias

Unfair patterns inherited from data.

What it is

Bias in AI means a system treats people or ideas unfairly in a consistent, repeatable way, usually because it learned a lopsided pattern from its data. It isn't a one-off mistake. A biased model makes the same kind of error again and again, and the errors tend to fall on the same groups of people.

The key word is "learned". Modern AI systems aren't programmed with rules like "rank men above women". They're trained on huge collections of examples, called datasets, and they pick up whatever patterns those examples contain. If the examples reflect an unequal past, the model learns the inequality along with everything else. If some people are barely present in the examples, the model gets worse at handling them. Nobody has to intend unfairness for it to end up in the system.

It helps to know that "bias" means three slightly different things in AI conversations:

Bias gets into a system at several points, and it's worth knowing them because each one needs a different fix:

Chatbots and image generators are built the same way, from vast amounts of text and pictures gathered from the internet in a process called pretraining. The internet is not a balanced portrait of humanity. It over-represents some languages, countries, and points of view, and it contains plenty of stereotypes. A language model absorbs all of that, which is why bias remains an active problem for every AI company, not a solved one.

Why it matters

AI now helps make decisions about people. Résumé screeners, credit models, insurance pricing, fraud flags, school admissions tools, medical triage, and police investigations all use automated scoring of some kind. When a system like that is biased, the cost isn't a slightly worse chatbot reply. It's a job interview that never happens, a loan that's refused, or a person arrested for something they didn't do.

It happens at scale and out of sight. A biased manager affects the people in front of them. A biased model can affect every applicant who passes through it, thousands a day, and each person only sees their own result. Nobody who was turned away can easily compare notes with everyone else, so patterns stay hidden until a journalist, a researcher, or a regulator goes looking.

It hides behind the look of objectivity. Because a model is maths, people tend to assume it's neutral. A score of 7 out of 10 feels more scientific than a human hunch, even when it's built on the same prejudices. This is why bias and automation bias make such a bad pair: the tool is skewed, and the people using it are inclined to trust it.

It's now a legal matter. Anti-discrimination laws already apply to decisions made with the help of software, and newer laws target AI directly. New York City's Local Law 144 requires employers who use automated tools in hiring or promotion to have the tool bias-audited within the past year, publish a summary of the audit, and tell candidates in advance; the city began enforcing it on 5 July 2023. A December 2025 audit by the New York State Comptroller found the city's enforcement falling short: its own review of 32 companies found one possible case of non-compliance where the auditors found at least 17, and the audit called the complaint process "ineffective". That shows how hard these rules are to police. In the European Union, the AI Act treats systems used in recruitment, credit scoring, education, and law enforcement as "high-risk", with requirements for data quality and human oversight. An amendment that entered into force in July 2026 pushed the date those rules apply to such systems back to 2 December 2027. Laws are changing quickly, so check the current position where you live and work.

It affects your own work with AI. Even if you never build a hiring system, you probably use AI to write job ads, draft marketing copy, generate images of "customers", or summarise feedback. Each of those can quietly carry stereotypes into your business. Knowing what bias looks like lets you catch it before your audience does.

Real-world examples

COMPAS: the risk score that started the fairness debate

On 23 May 2016, ProPublica published "Machine Bias", an investigation by Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner into COMPAS, a commercial tool from a company then called Northpointe. Courts and jails in the United States used it to score how likely a defendant was to commit another crime. The reporters obtained scores for more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014, then checked who was actually charged with new crimes over the following two years.

They found the tool was right about 61 percent of the time overall, and only 20 percent of the people it predicted would commit violent crimes went on to do so. The racial finding drew the most attention. Among defendants who did not go on to reoffend, Black defendants had been labelled higher risk nearly twice as often as white defendants: 44.9 percent against 23.5 percent. The error ran the other way for white defendants, who were more often labelled lower risk and then reoffended.

Northpointe rejected the analysis. Its argument was that the scores meant the same thing for both groups: a Black defendant and a white defendant with the same score reoffended at similar rates. Here's what made the case famous. Both sides were right about their own numbers. Within months, two separate research papers, one by Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan in September 2016 and one by Alexandra Chouldechova in October 2016, showed mathematically that when two groups have different underlying rates of the thing being predicted, a score cannot satisfy both definitions of fairness at once, except in unusual special cases. You have to choose which kind of fairness matters more, and that is a human and ethical decision, not a technical one.

A follow-up study by Julia Dressel and Hany Farid, published in Science Advances in January 2018, added another twist: COMPAS was no more accurate than predictions made by people with little or no criminal justice expertise, and a simple formula using just two facts about a defendant did about as well. The lasting lessons are that "is it fair?" needs a precise definition before anyone can answer it, and that a tool's reputation for sophistication tells you nothing about whether it deserves the trust placed in it.

The health algorithm that measured money instead of sickness

In October 2019, the journal Science published "Dissecting racial bias in an algorithm used to manage the health of populations" by Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. They studied a commercial algorithm that hospitals and insurers used to pick patients for extra-care programmes, the kind that assign a nurse to help someone with several chronic illnesses. It affected millions of patients.

The algorithm didn't use race at all. It predicted how much each patient's healthcare would cost in the coming year, on the reasonable-sounding idea that sicker people cost more. But in the US, less money is spent on Black patients than on equally sick white patients, largely because of unequal access to care. So the algorithm saw lower spending and concluded those patients were healthier. At the same risk score, Black patients were considerably sicker than white patients.

The researchers calculated that fixing this would raise the share of Black patients selected for extra help from 17.7 percent to 46.5 percent. They worked with the algorithm's maker to test a version that predicted health rather than cost alone. This is one of the clearest cases in the field, because it shows that bias can come from a single, quiet choice about what the model should predict, even when the data is accurate and the designers never touched race. It's often called a proxy problem: cost was standing in for need, and the stand-in was unequal.

Robert Williams and a face match that led to handcuffs

On 9 January 2020, Detroit police arrested Robert Williams on his front lawn in Farmington Hills, Michigan, in front of his wife and two young daughters. He was accused of stealing watches from a Shinola store in Detroit in 2018. The main lead against him was a facial recognition search that matched blurry surveillance footage to his old driver's licence photo. He spent the night in a jail cell, and the charges were later dropped.

His case became one of the first publicly known wrongful arrests in the US based on a facial recognition match. It connects to research showing that many face recognition systems perform unevenly across groups. In December 2019, the US National Institute of Standards and Technology (NIST) tested 189 algorithms from 99 developers and found that, when checking whether two photos show the same person, many produced false matches for Asian and African American faces between 10 and 100 times more often than for white faces, depending on the algorithm. NIST also found wide variation between products, and that some of the most accurate algorithms were also among the most even-handed, which means the problem can be reduced.

In June 2024, the City of Detroit settled Williams' lawsuit, brought with the ACLU. The city paid him $300,000 and agreed to rules the ACLU called the strongest of any US police department: officers can no longer arrest someone based only on a facial recognition result, and there must be independent evidence before a face match leads to a photo lineup. The ACLU noted that Detroit police had made three known wrongful arrests of this kind, and all three people were Black. The lesson reaches beyond policing: a biased tool does the most harm when people treat its output as proof rather than a lead to check.

Gemini's overcorrection: when fixing bias goes wrong

Image generators have a well-known habit of defaulting to stereotypes: ask for "a CEO" and older versions tended to produce mostly white men. Companies adjust their tools to show a wider range of people. In February 2024, Google's adjustment went too far. Users of Gemini's new image feature found it producing historically inaccurate pictures, such as the US Founding Fathers shown as Black, Asian, or Native American, and people of colour in Nazi-era uniforms when asked for "a 1943 German soldier".

On 22 February 2024, Google paused Gemini's ability to generate images of people. The next day, Prabhakar Raghavan, a Google senior vice president, published an explanation. He wrote that the tuning meant to make Gemini show a range of people had failed to account for cases that should clearly not show a range, and that the model had also become more cautious than intended and refused some harmless requests. In August 2024 Google began bringing the feature back, first for paying subscribers, using a newer image model.

This case matters because it shows that bias isn't only a problem of too little diversity. Crude fixes create new errors. A good system should show a range of people when the request is open ("a doctor") and follow the facts when the request is specific ("a 1943 German soldier"). Getting both right is hard, and there's no single switch labelled "unbiased".

The Apple Card inquiry: a case that found no violation

In November 2019, software entrepreneur David Heinemeier Hansson posted on Twitter that his new Apple Card credit limit was 20 times higher than his wife's, even though they filed joint tax returns and, by his account, her credit score was higher. Apple co-founder Steve Wozniak posted that he'd had a similar experience. New York's Department of Financial Services (DFS) opened an investigation into Goldman Sachs, the bank behind the card.

The ending is often misremembered. In March 2021, after reviewing records for about 400,000 New York applicants, DFS found no violation of fair lending laws. It did criticise poor customer service and a lack of transparency, which had undermined trust, and it said credit scoring and anti-discrimination law needed strengthening, because traditional measures of creditworthiness can carry forward historical inequality. Many couples had also assumed that sharing an account meant sharing the same credit terms, which wasn't how the card worked.

It's included here because honest thinking about bias needs cases that didn't pan out as well as ones that did. A striking anecdote is a reason to investigate, not proof. And a system that can't explain its decisions will be suspected of bias whether or not it is biased, which is a strong argument for transparency in its own right.

Try it yourself

You don't need any technical skills for these tests. Use any chatbot or image generator you have access to. Results change as companies update their models, so treat what you find as a snapshot, not a verdict.

  1. Run the "default person" test on an image tool. Ask for single images with open prompts: "a photo of a CEO", "a photo of a nurse", "a photo of a software engineer", "a photo of a cleaner". Generate four images for each. Note who appears by apparent gender, age, and skin tone. Then try "a photo of a person" and see who the tool thinks a default person is.
  2. Run the history test. Ask the same tool for something specific, such as "a group photo of the US Senate in 1920". Does it follow the historical record, or does it change it? A well-tuned tool should vary people when the prompt is open and follow the facts when it isn't.
  3. Try the name-swap test on a chatbot. This is how researchers check hiring tools. Write a short, identical job application twice, changing only the name, for example "Emily Walsh" and "Lakisha Washington", or "James" and "Priya". Paste each into a fresh chat with the same prompt: “ You are screening applicants for a junior marketing role. Rate this application from 1 to 10 and give one sentence of reasoning. [paste application here] “ Run each version three times, because answers vary. If the scores differ consistently while the text is identical, you've found a bias. If they don't, that's worth knowing too.
  4. Test the stereotype completions. Ask a chatbot to "write a two-sentence story about a nurse and a surgeon", then "a two-sentence story about a pilot and a flight attendant". Which pronouns does it give each person? Ask it to write ten short stories and count.
  5. Test other languages and places. Ask the same practical question, say about starting a small business, as if you lived in the United States and then as if you lived in Nigeria or Indonesia. Is the advice equally specific and useful, or does one version fall back on generalities?
  6. Ask the model to check itself. After any draft that describes people, paste it back with this prompt: “ Review the text below for stereotypes or assumptions about gender, race, age, disability, nationality, or class. For each one, quote the phrase, say what assumption it makes, and suggest a neutral rewrite. Don't flag anything that is a genuine fact the text needs. [paste text here] “ The model won't catch everything, and it can over-flag, so read its suggestions critically.

To reduce bias in your own everyday use:

Common mistakes and misunderstandings

"It's just maths, so it's neutral." The maths faithfully reproduces whatever pattern is in the data and whatever goal it was given. Neutral maths plus unequal data gives unequal results.

"If we remove race and gender from the data, the bias goes away." The health algorithm never used race. Amazon's screener picked up gender from words like "women's". Other details, such as postcode, school, hobbies, or spending, often stand in for the characteristic you removed. These are called proxies, and they're the main reason "just delete the column" rarely works.

"There's one correct definition of fair." The COMPAS debate proved otherwise. Equal error rates, equal meaning of scores, and equal selection rates are all reasonable goals, and they often can't be met at the same time. Choosing between them is a values decision that people, not software, have to make and explain.

"Bias only matters in serious systems like policing." Everyday content carries it too. Marketing images that always show the same kind of person, job ads written in language that puts some applicants off, and chatbot summaries that treat one group's complaints as less serious all add up.

"More diversity in the output fixes it." Gemini showed that forcing variety everywhere creates new errors. Good fixes depend on context: represent people fairly when the request is open, and respect the facts when it isn't.

"A viral anecdote proves the system is biased." The Apple Card story shows why investigations matter. One couple's experience is a reason to look closely; it isn't the finding. Equally, one clean test on your own chatbot doesn't prove it's unbiased.

"Bias is fixed once and stays fixed." Models are updated, used on new people, and applied to new tasks. A tool audited last year may behave differently today, which is why the New York law requires audits to be recent. Keep testing, the way you would with evals for any prompt.

"Humans are unbiased, so let's keep humans instead." People are biased too, often in ways that are harder to measure. The honest goal is not a perfect judge but a process where both people and tools are checked, errors can be spotted, and decisions can be explained and challenged.

Related elements