Aiodic

Back to the table

55 Advanced Dl Deep Learning Machine Learning

Deep Learning

Machine learning with many-layered neural networks.

What it is

Deep learning is machine learning done with neural networks that have many layers stacked one after another. A neural network is a program made of simple calculating units whose connection strengths, called weights, are adjusted during training until the program gives good answers. "Deep" means nothing more mysterious than lots of layers between the input and the answer. A shallow network might have one or two; a deep one has dozens, and the largest modern systems have more than a hundred.

That sounds like a small change of recipe, and in a sense it is. What makes it matter is what the layers do. In a deep network, each layer takes the output of the layer before and builds something slightly more complex from it. Nobody tells the layers what to look for. They work it out during training, because building up in stages turns out to be the most efficient way to get the answers right.

The classic illustration is a network that recognises photos. Picture the stages:

  1. The first layers respond to tiny, simple patterns: an edge running this way, a patch of colour, a change from light to dark.
  2. The next layers combine those edges into slightly bigger shapes: corners, curves, stripes, a small circle.
  3. Middle layers combine shapes into textures and parts: fur, an eye, a wheel, the petal of a flower.
  4. The last layers combine parts into whole things: a cat's face, a bicycle, a sunflower. The final layer turns that into an answer, such as "this is a cat".

Three of the field's founders, Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, described deep learning in a 2015 review in the journal Nature as models made of multiple processing layers that learn representations of data at multiple levels of abstraction. "Representation" here just means the way the network describes what it's looking at, in its own internal numbers. "Levels of abstraction" is the edges, shapes, parts, and objects ladder above.

The same layering happens with sound and text, even though it's harder to picture. In a speech recogniser, early layers pick up bursts of sound, later ones pick up the sounds that make up syllables, and later still whole words. In a language model, early layers deal with word pieces and grammar, and later layers with meaning, tone, and what the sentence is about. That's the popup's line about going "from edges in images to meaning in sentences".

How it differs from older machine learning

Before deep learning, people building a photo classifier or a speech system spent much of their effort on "feature engineering": experts deciding by hand which measurements to feed the computer. For a face detector, that might be the ratio of the distance between the eyes to the width of the mouth. The learning algorithm then weighed up those hand-picked measurements.

Deep learning hands that job to the layers. You give the network raw material, the pixels of a photo or the waveform of a recording, and the early layers learn their own features. That's the real shift. It's why the same basic approach works on photos, sound, text, protein structures, and game boards, and why a field that used to need a different specialist for each problem now shares one toolkit.

The pieces you'll hear about

Most of the named designs in AI today are kinds of deep network:

They differ in how the layers are wired. They share the core idea: many layers, each building on the last, all trained together from examples.

Why it matters

It's the reason AI suddenly got good. For decades, computers were poor at seeing, hearing, and reading. From about 2012, deep learning brought step changes in speech recognition, image recognition, and translation within a few years, and it underpins nearly every AI product you use now, from voice assistants to chatbots. If you want to understand the modern AI boom, this is the element that started it.

It explains why AI needs so much data. A deep network has millions or billions of weights, the parameters, and each one has to be set by examples. That's why datasets became so valuable, and why the biggest models are trained on huge chunks of the internet during pretraining. Deep learning is hungry, and its hunger shapes the whole industry.

It explains why AI needs so much computing power. Training a deep network means doing the same simple arithmetic trillions of times. The chips that do that fastest, originally built for video games, have become some of the most sought-after products in the world, as the second example below shows.

It explains why AI can be strange. Because a deep network learns its own features, it can learn features nobody intended: a shortcut in the data, a background pattern, a quirk of the photos it was trained on. That's why a system can be superbly accurate in testing and then fail in ways no person would, and why tiny, invisible changes to a photo can fool it. You'll meet both problems below.

It explains why AI is hard to explain. The knowledge in a deep network is spread across its layers as numbers. We can see what some parts respond to, but nobody can yet read a full account of why a large model gave a particular answer. That gap is what the field of interpretability is trying to close.

Real-world examples

Deep learning gets its name, and speech recognition jumps (2006 to 2012)

Neural networks with several layers existed in the 1980s, but they were hard to train: the more layers you added, the less the early layers learned. In July 2006, Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh published "A Fast Learning Algorithm for Deep Belief Nets" in the journal Neural Computation. It showed a way to train a deep network one layer at a time, each layer learning to describe the output of the one below, and then fine-tune the whole stack. The paper is widely credited with reviving interest in deep networks, and "deep learning" became the name for the approach.

Speech was the first field to be transformed. A University of Toronto news story in November 2012 describes how Hinton's students Abdel-rahman Mohamed and George Dahl presented deep network speech results at a 2009 conference, were invited to intern at Microsoft Research, and applied the method to much larger speech tasks there. Another student, Navdeep Jaitly, worked with Google to build it into its system, and by 2012 Google was using a deep neural network for voice search in Android 4.1.

The public moment came on 25 October 2012 in Tianjin, China. Rick Rashid, Microsoft's chief research officer, gave a keynote at the Computing in the 21st Century conference. As he spoke in English, a deep learning speech recogniser turned his words into text, a translation system converted them to Chinese, and a speech system spoke them aloud in Mandarin in a voice modelled on his own. The audience of students applauded. Rashid wrote afterwards that the deep network approach, developed with the University of Toronto, had cut the word error rate by over 30 percent compared with previous methods: instead of roughly one word in four or five being wrong, it was now about one in seven or eight.

The lesson: depth wasn't a new idea in 2006. What changed was finding practical ways to train it, and then showing, on a stage, that it worked better than decades of hand-built methods.

Graphics chips, AlexNet, and Nvidia (2012 to 2025)

Why did deep learning take off around 2012 and not in 1992? Three things arrived together. The first was data: the ImageNet collection, which the AlexNet paper describes as over 15 million labelled images in over 22,000 categories. The second was better techniques, such as simpler activation functions (the switch inside each unit that decides how strongly to pass a signal on) that let deep networks train faster, and "dropout", which randomly switches off units during training so the network can't lean on any one of them. The third was cheap, fast computing.

The AlexNet paper shows all three in one place. The Toronto team, Alex Krizhevsky, Ilya Sutskever, and Hinton, trained a network with 60 million parameters and 650,000 units on 1.2 million ImageNet photos. It took five to six days on two Nvidia GTX 580 graphics cards, the kind sold to people who play video games. They wrote that their results could be improved simply by waiting for faster graphics chips and bigger datasets. Its win in the 2012 ImageNet contest is told in Artificial Intelligence.

Graphics chips suit deep learning because both involve the same kind of work: huge numbers of simple multiplications done at the same time. A game draws millions of pixels in parallel; a deep network multiplies millions of weights in parallel. Nvidia bet on that match, and deep learning made it one of the most valuable companies ever. On 9 July 2025 it became the first company to touch 4 trillion dollars in market value during trading, and on 29 October 2025 it became the first to close above 5 trillion. Market values move every day, so treat these as dated milestones, not the current figure. The point stands: the hardware that made deep learning possible became an industry of its own.

Looking inside the layers (2013 to 2015)

The edges-to-objects story above isn't just a teaching picture; researchers have seen it. In November 2013, Matthew Zeiler and Rob Fergus of New York University published "Visualizing and Understanding Convolutional Networks". They built a way to project what a unit in a middle layer was responding to back into picture form. The images showed exactly the ladder: early layers picking up edges and colour blobs, later layers picking up textures and patterns, and the top layers responding to object parts and whole objects, such as dogs' faces. They also used what they saw to spot weaknesses in AlexNet's design and build a model that beat its results on ImageNet. Looking inside wasn't only for curiosity; it made the networks better.

In June 2015, three Google engineers, Alexander Mordvintsev, Christopher Olah, and Mike Tyka, published a blog post called "Inceptionism: Going Deeper into Neural Networks". They ran a trained image network backwards: instead of asking "what's in this picture?", they asked the network to change the picture to show more of whatever it was detecting. Lower layers filled images with strokes and ornament-like patterns. Higher layers turned clouds into birds, filled horizons with towers and pagodas, and made birds and insects appear among leaves. When Google released the code on 1 July 2015 under the name DeepDream, the dreamy, slightly unsettling images spread across the internet.

One image in the post is worth remembering. When they asked the network to draw dumbbells, it drew dumbbells with muscular arms attached. The engineers suggested the network had rarely seen a dumbbell without someone lifting it, so it had learned "arm" as part of "dumbbell". That's deep learning in a single picture: it learns whatever pattern in the data gets the answer right, including patterns people never meant to teach.

Panda, gibbon, and the stop sign (2013 to 2018)

In December 2013, a team including Google researcher Christian Szegedy posted a paper with the modest title "Intriguing properties of neural networks". It reported something alarming. You could take a photo the network classified correctly, change the pixels by an amount too small for a person to notice, and the network would confidently get it wrong. These altered images came to be called adversarial examples. Worse, an image crafted to fool one network often fooled others trained separately.

A year later, Ian Goodfellow, Jonathon Shlens, and Szegedy published the picture that made the problem famous. A photo of a panda, which Google's GoogLeNet image network labelled "panda" with 57.7 percent confidence, had a faint layer of what looks like static added. To a person, it's still obviously a panda. The network now said "gibbon", with 99.3 percent confidence.

Then the problem left the screen. In a paper first posted in July 2017 and presented at the CVPR conference in 2018, a team including researchers from the University of Michigan, the University of Washington, and UC Berkeley put a few black and white stickers on a real stop sign. A road-sign classifier read it as a "Speed Limit 45" sign in every image in their lab tests, and in 84.8 percent of video frames filmed from a moving car. The stickers were designed to look like ordinary graffiti or art.

The lesson isn't that deep learning is useless; it's that a network's accuracy on test photos doesn't mean it sees the way you do. It can rely on features no person would use, and someone who understands that can exploit it. Systems that matter need checks around them, which is what guardrails are for.

The electricity bill (2019 to 2021)

In June 2019, Emma Strubell, Ananya Ganesh, and Andrew McCallum of the University of Massachusetts Amherst published "Energy and Policy Considerations for Deep Learning in NLP", presented at the ACL conference that July. They estimated the energy and cost of training several language models. The headline figure was for an automated search that trains many versions of a transformer to find the best design: about 626,000 pounds of carbon dioxide equivalent, roughly five times the lifetime emissions of an average American car, including its fuel, by the paper's own comparison table.

That number was widely quoted, and it was challenged. In April 2021, a team led by David Patterson of Google and the University of California, Berkeley, published "Carbon Emissions and Large Neural Network Training". Using Google's actual measurements, they said the 2019 figure for that search was 88 times too high. The same paper estimated that training OpenAI's GPT-3 used about 1,287 megawatt-hours of electricity and produced about 552 tonnes of carbon dioxide equivalent, and argued that choosing efficient models, data centres, and chips can cut emissions by 100 to 1,000 times.

Both papers agree on the point that matters for you: deep learning's energy use is real and large, it depends heavily on how and where the training is done, and the only way to know is for companies to measure and publish it. Running a model once it's trained, called inference, has its own energy cost, which adds up over billions of uses.

An everyday scenario: the plant nursery's photo tool

Imagine Priya, who runs a small plant nursery and uses a phone app that identifies plants from photos. It's excellent: point it at a leaf and it names the species in a second. Under the hood, a deep network is climbing the ladder, from leaf edges, to vein patterns and leaf shapes, to "this is a Japanese maple".

One week a customer brings in a houseplant photographed on a patterned tablecloth, and the app confidently names a plant it isn't. Priya tries again with the plant against a plain wall, and it gets it right. Nothing is broken. The network learned from photos, and some of the features it learned belong to the kind of background those photos had, not only to the plant. Knowing that, Priya takes photos against a plain background, checks the app's answer against the leaf herself for anything she'll sell, and treats a confident answer as a strong hint rather than a certainty.

Try it yourself

You can look inside a real deep learning image classifier in your browser, free and without an account. Allow about 30 minutes.

  1. Open CNN Explainer at poloclub.github.io/cnn-explainer. It was built by researchers at Georgia Tech and Oregon State University to teach how image networks work. It runs a small convolutional network called Tiny VGG that sorts pictures into ten classes, such as ladybug, pizza, school bus, koala, and espresso. It works best on a laptop or desktop.
  2. Pick an image. Along the top is a row of small pictures. Click the ladybug. On the far left you'll see it split into its red, green, and blue channels: that's the raw input, just numbers for each pixel.
  3. Follow the layers left to right. Each column of small squares is one layer, and each square shows what one unit in that layer responds to. Early columns look like the original picture traced in edges and light and dark patches. Later columns look more abstract. That's the ladder from simple features to complex ones, happening in front of you.
  4. Click a square in the first layer marked "conv". The tool zooms in and shows the small pattern detector, called a kernel, sliding across the image. This is the whole trick of a convolutional network: the same small detector checked at every position.
  5. Look at the end. The last column gives a score for each of the ten classes. Try each sample picture and see which the network finds easy and which it hesitates over.
  6. Try your own picture. Click the plus icon at the end of the row of sample images to add a picture of your own. Try something that belongs to one of the ten classes, and something that doesn't, such as a photo of your shoe. The network still has to pick one of its ten answers, and it may pick one confidently. That's a small version of a real limit: a classifier only knows the categories it was trained on.
  7. Play one round of Quick, Draw! at quickdraw.withgoogle.com. It's a free Google game where a neural network tries to guess your doodle as you draw it. Notice how it often guesses right from just a few strokes, the network picking up on simple features first, and how a strange drawing can send it down the wrong path.
  8. Ask a chatbot to explain what you saw. Paste this into any AI chatbot:
I just used CNN Explainer to look inside a small image-recognition
network. Early layers looked like edges and light and dark patches,
and later layers looked more abstract. In plain English, with no
maths, explain why a network that stacks many layers learns better
features than one with a single layer. Give me one everyday analogy,
and then tell me one thing deep networks are bad at.
  1. Notice the deep learning around you. For a day, note each time your phone recognises your face, transcribes a voice message, finds a photo by what's in it, or translates a sign. Each one is a deep network climbing its own ladder of features.

Common mistakes and misunderstandings

"Deep means the AI understands deeply." The word refers to the number of layers, not to the depth of understanding. A deep network can be brilliant at a task and still have no idea what a panda is in the way you do, as the gibbon example shows.

"Deep learning is a different thing from neural networks." It isn't. Deep learning is neural networks with many layers, trained with the same basic method of guessing, measuring the error, and adjusting the weights. The name marks a change in scale and practice, not a new kind of machine.

"More layers is always better." Adding layers helps only if you have the data, the computing power, and the training techniques to make use of them. For many small business problems, such as predicting sales from a spreadsheet, simpler machine learning often does as well, costs far less, and is easier to explain.

"A high accuracy score means it sees like a person." A deep network can reach high accuracy by relying on features no person would use, such as a background, a watermark, or a pattern of pixels. Test any AI tool on your own real cases, including awkward ones, before trusting it.

"You need your own giant dataset to use deep learning." Most people and businesses never train a deep network from scratch. They use one someone else has trained, through an app or an API, or adapt an existing model to their task with a little of their own data, a technique called transfer learning.

"We know what the layers learned because we designed them." People design the shape of the network, not what it learns. Visualisations show us what some units respond to, but a full explanation of a large network's decisions is still out of reach. Be wary of any tool that claims to tell you exactly why its AI decided something.

"Deep learning is new." The ideas go back decades, the name caught on around 2006, and the breakthroughs came around 2012. What's new since then is the scale of data, chips, and money. Knowing the history helps you read the headlines calmly, in both directions.

"Training is the only energy cost." Training a big model uses a lot of electricity once. Answering questions uses a smaller amount each time, but across billions of uses it adds up. Both count.

Related elements