Aiodic

Back to the table

5 Beginner Nn Neural Network Core Concepts

Neural Network

Layers of connected math units that learn patterns.

What it is

A neural network is a way of building a computer program that learns from examples instead of following rules someone wrote by hand. It's made of many small, simple calculating units, usually called neurons or nodes, connected to each other in layers. Each connection has a strength, a number called a weight, that decides how much one unit influences the next. Training a network means adjusting those weights, a tiny amount at a time, until the network's answers match the examples it's shown.

That's the whole idea, and it's worth saying plainly because the name makes it sound more mysterious than it is. There are no wires or cells inside your computer that resemble a brain. A neural network is arithmetic: multiply, add, and pass the result along. What makes it powerful is the scale and the learning, not any single part.

Here's how one unit works. It receives some numbers from the units before it. It multiplies each number by the weight of that connection, adds them up, and then applies a simple rule that decides how strongly to pass the result on. The technical name for that rule is an activation function, which just means a switch that lets strong signals through and quietens weak ones. The output then becomes an input to units in the next layer.

Stack those units into layers and you get the familiar shape:

The "loosely inspired by the brain" part is real but thin. In the brain, neurons pass signals to each other through connections called synapses, and learning partly works by making some of those connections stronger or weaker. The Nobel Committee for Physics used exactly this comparison when it described the field in 2024: nodes stand in for neurons, and connections that can be made stronger or weaker stand in for synapses. Beyond that, the resemblance fades. Real neurons are far more complicated, and the brain doesn't learn the way a computer network does.

How it learns

Learning happens in a loop, and it's the same loop whether the network has ten weights or hundreds of billions.

  1. Show the network an example, such as a photo labelled "cat".
  2. Let it make a guess. At first the weights are random, so the guess is rubbish.
  3. Measure how wrong the guess was. This measure is called the loss, or the error.
  4. Work backwards through the network and figure out, for every single weight, whether nudging it up or down would have made the error a little smaller. This step is called backpropagation, short for "backward propagation of errors".
  5. Nudge every weight a small step in the helpful direction. This is called gradient descent: "gradient" means the slope of the error, and "descent" means walking downhill on it.
  6. Repeat, millions of times, with millions of examples.

No single nudge teaches the network anything you'd recognise. But after enough rounds, the weights settle into a pattern that turns inputs into good outputs, including for examples the network has never seen before. That last part, called generalisation, is the point. A network that only memorises its training examples is useless; one that has picked up the underlying pattern can handle new ones.

The weights are what the network knows. When people talk about a model's parameters, they mostly mean these weights. The 1989 network in the second example below had fewer than ten thousand. OpenAI's GPT-3, described in a 2020 paper, had 175 billion. For a sense of scale, a careful 2009 count of adult men's brains estimated that a human brain has about 86 billion neurons, though a weight in a computer and a neuron in a head are not comparable things.

Where it fits

Neural networks are one approach within machine learning, the wider field of programs that learn from data. When a network has many hidden layers, the approach is called deep learning, and "deep" simply refers to the number of layers. Almost everything people now call AI, from photo search on your phone to chatbots and image generators, runs on deep neural networks. A transformer, the design behind today's language models, is one particular way of wiring a neural network. A diffusion model, behind many image generators, is another.

Why it matters

It's the engine under almost every AI tool you use. When a chatbot answers a question, a neural network is predicting the next token, over and over. When your phone groups photos by face, or a transcription app turns speech into text, or an image tool paints what you described, a neural network is doing the work. Understanding the basic idea, weights adjusted by examples, explains a surprising amount of how these tools behave.

It explains why AI is good at fuzzy tasks and odd at precise ones. A neural network is a pattern-matcher. It's excellent where the rules are too complicated to write down, such as recognising a face in poor light, or judging whether a sentence sounds natural. It can be unreliable where one exact answer matters and the pattern is thin, such as an obscure date, a precise calculation, or a citation. That is part of why language models sometimes produce confident, fluent, wrong answers, a failure known as hallucination.

It explains why data matters so much. A network learns only what its examples contain. If the datasets it trains on are skewed, the network learns the skew along with everything else. This is the root of most bias problems in AI. Nobody programmed the prejudice; the weights absorbed it from the examples.

It explains why AI is hard to inspect. In a normal program, you can read the rules. In a neural network, what it "knows" is spread across millions or billions of numbers, none of which means anything on its own. Asking why a network gave a particular answer is a genuinely hard research question, and the field that studies it is called interpretability.

It explains the cost. Training means running that guess, measure, and nudge loop over enormous amounts of data, and bigger networks need more of everything: more examples, more computer chips, more electricity, and more time. That's why the largest models are built by a handful of well-funded companies, and why pretraining a model from scratch is out of reach for most businesses, while fine-tuning an existing one is not.

Real-world examples

The Perceptron: a "Navy device" that learned (1958)

In July 1958, the U.S. Office of Naval Research held a demonstration of the perceptron, a learning machine designed by Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory. The perceptron was a single-layer neural network, which means it had only one layer of adjustable weights, and its job was to sort inputs into one of two categories. In the demonstration, an IBM 704 computer running the idea learned to tell left from right after about fifty tries.

The press loved it. On 8 July 1958, The New York Times ran the story under the headline "New Navy Device Learns by Doing", and reported that the Navy expected later versions to walk, talk, see, write, reproduce themselves and be conscious of their existence. The New Yorker went further still, calling it a serious rival to the human brain. Rosenblatt went on to build the Mark I Perceptron, a physical machine with its own light sensors acting as an eye, which is now in the collection of the Smithsonian Institution.

The promises ran far ahead of the machine. In 1969, Marvin Minsky and Seymour Papert of MIT published a book called "Perceptrons" that set out, with mathematical proofs, what a single-layer network could not do. The book is often blamed for drawing money and attention away from neural networks for years afterwards. Rosenblatt drowned in 1971, on his 43rd birthday, long before his idea was vindicated.

Two lessons from 1958 still apply. First, the basic idea of a neural network, simple units with adjustable weights that learn from examples, is almost seventy years old; what changed is scale. Second, the gap between a real demonstration and the headlines about it is not new. When you read that an AI system "thinks" or "understands", remember the machine that learned left from right and was expected to become conscious.

Backpropagation, the zip-code reader, and 33 years of progress (1986 to 2022)

The fix for the perceptron's limits was to add hidden layers, but that raised a hard problem: if the network gets an answer wrong, how do you know which of the hidden weights to blame? In October 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published a short paper in Nature titled "Learning representations by back-propagating errors". It showed that backpropagation, the method of working backwards from the error to adjust every weight, could train networks with hidden layers, and that those hidden units would come to represent useful features of the task on their own. Similar mathematics had appeared earlier, but this paper is the one that made the technique famous and set off a wave of research.

Three years later came one of the first practical uses. In December 1989, Yann LeCun and colleagues at Bell Labs published "Backpropagation Applied to Handwritten Zip Code Recognition" in the journal Neural Computation. Their network learned to read handwritten digits from real zip codes supplied by the U.S. Postal Service. It was a kind of network now called convolutional, meaning it scans the image with small pattern detectors, the same idea later used in AlexNet in 2012 and in the image recognition on your phone today. By 1998, LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner reported that a check-reading system built on these networks was deployed commercially and read several million checks per day.

The best way to feel how much has changed is a reproduction published in 2022 by the AI researcher Andrej Karpathy. He rebuilt the 1989 network from the paper. By his account, the original had 9,760 weights, learned from 7,291 small grey images of digits, and took three days to train on a Sun workstation. His version trained on an ordinary laptop in about 90 seconds and gave a similar error rate. Nothing about the basic recipe had changed; the computers had. That, more than any new idea, is the story of neural networks between 1989 and today: the same loop of guess, measure, and nudge, run on vastly more data with vastly more computing power.

The prizes that confirmed it (2019 and 2024)

On 27 March 2019, the Association for Computing Machinery named Yoshua Bengio, Geoffrey Hinton, and Yann LeCun as the recipients of the 2018 Turing Award, often described as computing's highest honour, for breakthroughs that made deep neural networks a critical part of computing. All three had kept working on neural networks through the long years when much of the field considered them a dead end.

Then, on 8 October 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to John Hopfield of Princeton University and Geoffrey Hinton of the University of Toronto, "for foundational discoveries and inventions that enable machine learning with artificial neural networks". Hopfield's network, from the early 1980s, could store patterns and rebuild a whole image from a damaged or incomplete one, using mathematics borrowed from the physics of magnetic materials. Hinton built on it to create the Boltzmann machine, a network that learns the characteristic features of a set of data. It was a striking choice: a physics prize for the ideas underneath modern AI.

There's a twist that belongs in the story. In May 2023, Hinton left his job at Google and said publicly that he had done so in order to speak freely about the risks of AI, including misinformation and the loss of jobs. One of the people most responsible for making neural networks work is now among the loudest voices asking for care in how they're used. That debate is covered in alignment.

An everyday case: the photo search that knows what a beach is

Imagine Tom, who sells handmade surfboards and has twelve thousand photos on his phone. A customer asks for a picture of a particular board in the water, and Tom types "beach" into his photo app's search box. Up come hundreds of beach photos, even though he never tagged a single one.

What's happening is a neural network at work. The app's makers trained a network on a huge number of labelled images, and its hidden layers learned, without anyone describing them, the features that tend to appear in beach photos: a horizon line, the texture of sand, a particular band of blue. When Tom's photos pass through that network, each one gets scored against thousands of such concepts, and the search simply returns the high scorers.

The same scenario shows the limits. A photo of a sandy car park at dusk might sneak in, because it shares the features the network latched onto. A beach photo in heavy fog might be missed, because it doesn't look like most beach photos the network saw. The network isn't checking a definition of "beach". It's matching patterns, and it's usually right, which is exactly how neural networks behave everywhere else too.

Try it yourself

You can watch a real neural network learn in your browser, without code or an account. Allow about half an hour.

  1. Open the TensorFlow Playground at playground.tensorflow.org. It's a free teaching tool. You'll see dots in two colours on the right, a network in the middle, and controls along the top.
  2. Pick the easiest data. Under "Data" on the left, choose the pattern where the two colours sit in two separate clumps. Press the play button at the top left. Within seconds the background shades into two regions that separate the colours. The number labelled "Test loss" falls toward zero. That falling number is the error, and you're watching gradient descent reduce it.
  3. Look at the weights. The lines between the neurons are the connections. Their thickness shows how strong each weight is, and the colour shows whether it's positive or negative. Watch them change as training runs. That changing is the learning.
  4. Make it hard. Press the reset button, then choose the spiral pattern, where the two colours wind around each other. Remove hidden layers until there are none, and press play. The network can only draw a straight line, and it fails. This is the limit of the 1958 perceptron.
  5. Add depth. Add two or three hidden layers with several neurons each, and press play again. Give it a minute or two. The network gradually carves out something close to a spiral; if it struggles, add more neurons to each layer, or tick one or two of the extra input boxes under "Features" on the left, which hand the network ready-made curved shapes to build from. Hover over individual neurons to see the simple shapes each one has learned, and notice how later layers combine earlier shapes into more complicated ones. That's what "hidden layers learn features" means.
  6. Break it on purpose. Change the "Learning rate" at the top to a much larger number. The loss jumps around and may never settle, because each nudge to the weights is too big. Set it very small, and learning becomes painfully slow. Choosing that step size well is a real part of training networks.
  7. Ask a chatbot to explain what you saw. Paste this into any AI chatbot:
I just watched a neural network learn in the TensorFlow Playground.
It failed on the spiral pattern with no hidden layers, but succeeded
with three hidden layers. Explain in plain English, with no maths
and no jargon, why the extra layers made the difference. Then give
me one everyday analogy.
  1. Spot the networks around you. For the next day, notice each time a tool recognises a face, transcribes your voice, suggests the next word as you type, or finds a photo by what's in it. Each of those is a trained neural network, doing a version of what you just watched on a much larger scale.

Common mistakes and misunderstandings

"It works like a brain." The inspiration was the brain, and the vocabulary (neurons, synapses) comes from biology. But an artificial neuron is a line of arithmetic, and a network learns by a mathematical procedure the brain almost certainly doesn't use. Treat "inspired by the brain" as history, not as a description of how it works.

"Someone programmed it to know that." Nobody wrote the rules a neural network follows. People chose the design, the data, and the training procedure, and the weights set themselves. That's why a network can pick up patterns its makers never intended, both good and bad.

"Bigger is always better." More weights let a network learn more complicated patterns, and in recent years size has driven much of the progress. But a bigger network also needs more data, costs more to run, and can simply memorise its examples if the data is thin. For many practical jobs, a smaller network trained on the right examples beats a giant one.

"It looks things up." A trained network doesn't search a database of its training examples. What it learned is compressed into its weights, which is why it can answer questions it has never seen and also why it can misremember facts. Tools that need to quote accurate, current information add a separate retrieval step, such as RAG, on top of the network.

"If it's accurate on average, it's accurate for me." A network is only as good as the match between its training data and your situation. A model trained mostly on one kind of photo, voice, or writing can do worse on others. Test it on your own real cases before relying on it.

"Neural networks are brand new." The perceptron was demonstrated in 1958, backpropagation made famous in 1986, and a network was reading zip codes in 1989. What's new is the scale of data and computing power. Knowing this protects you from hype in both directions: the idea isn't magic, and it isn't a passing fad.

"The weights can be read like a recipe." You can open a network and see every number, but the numbers don't explain themselves. Understanding why a network made a specific decision is an unsolved research problem, so be wary of any tool that claims to give you a full explanation of its reasoning.

Related elements