Aiodic

Back to the table

16 Beginner Df Diffusion Model Core Concepts

Diffusion Model

A model that creates images by removing noise step by step.

What it is

A diffusion model is a kind of AI that makes pictures by starting with pure static and cleaning it up, one small step at a time, until an image appears. If you have ever typed a sentence into an AI image tool and watched a blurry smear sharpen into a finished picture, you have probably watched a diffusion model at work.

The easiest way to understand it is to picture the process running backwards. Take a photograph of a cat. Sprinkle a little random noise over it, like the grain on an old film photo. Then add a bit more, and a bit more, until after many rounds the cat has vanished completely and all that's left is television snow. That's the "diffusion" part of the name: the picture slowly dissolves, the way a drop of ink spreads out in a glass of water until the water is evenly grey.

Now the clever part. During training, the AI is shown millions of these half-dissolved pictures and asked one simple question each time: what noise was added here? Guess the noise, and you can subtract it, which leaves a slightly cleaner picture. The model never learns to paint a cat in one go. It learns to take a slightly messy picture and make it slightly less messy. That turns out to be enough, because if you repeat that small clean-up many times, you can start from pure static and end with a picture that never existed before.

A useful everyday comparison is a sculptor who works in fog. At first the sculptor can see nothing but a grey blur. They make a guess about what's in there and carve a little. The shape gets clearer, so the next guess is better, and the next carve more precise. After enough passes, a statue stands in a clear room. A diffusion model does the same thing with pixels.

So where does your prompt come in? The words you type are turned into numbers (the same kind of meaning-map described under embeddings), and at every clean-up step the model is nudged to remove noise in a way that moves the picture towards those words. Without a prompt, a diffusion model would still produce a picture, just a random one. With a prompt such as "a watercolor lighthouse at dawn", every step leans a little towards lighthouse, a little towards watercolor, a little towards the soft light of early morning.

A few terms you will meet in image tools come straight from this process:

Video works the same way with one more dimension. Instead of cleaning up one frame of static, a video diffusion model cleans up a whole block of frames at once, so that each frame agrees with the ones before and after it. That's much harder, which is why AI video lagged a couple of years behind AI images.

Under the hood, a diffusion model is a neural network trained through deep learning on huge collections of captioned pictures. Many recent ones use the transformer design, the same wiring found in chatbots, as the part that predicts the noise. Many also work on a compressed version of the image rather than on every pixel, an idea from a 2021 research paper on "latent diffusion" that made the approach cheap enough to run on an ordinary gaming computer. Some newer tools use close cousins of diffusion with names like "flow matching", which follow the same noise-to-picture idea by a more direct route.

Why it matters

It is the engine behind the AI image boom. Most of the image and video tools creators use, including Stable Diffusion, Flux, Adobe Firefly and many others, are built on diffusion or its close relatives. Knowing how the engine works explains most of their quirks. Not every tool works this way, though. When OpenAI added image generation to its GPT-4o model in March 2025, its own safety document described the new system as autoregressive, meaning it builds the picture piece by piece, the way a chatbot writes a sentence, and contrasted it with its earlier DALL·E models, which were diffusion models.

It explains why the same prompt gives different pictures. Beginners often think the tool is being moody. It isn't. Each attempt starts from different random static, so each ends up somewhere different. Once you know about seeds, you can lock in a picture you like and change one word at a time to see what that word really does.

It explains the classic flaws. Because the model is cleaning up noise towards "what pictures like this usually look like", it's excellent at textures, lighting and overall mood, and historically weaker at things that need exact counting or strict rules: the number of fingers on a hand, the spelling of words on a sign, the correct number of legs on a table. Newer models have improved a lot on all three, but the underlying reason is worth knowing. The model has learned what's typical, not what's correct.

It raises the biggest legal questions creators face. A diffusion model learns what things look like from millions of captioned pictures, often scraped from the public web. Stable Diffusion, for example, was trained on material from a huge public dataset of web images (the story of what that dataset turned out to contain is told under datasets). That raises two questions that are still being argued in courts and copyright offices: was it lawful to train on those pictures, and who owns what comes out? Both affect anyone who sells AI-made work.

It changes what "making an image" costs. A picture that once needed a photographer, a location, or hours of illustration can now be drafted in seconds. That's a gift for mock-ups, book-cover concepts, social posts and storyboards. It's also why illustrators and photographers have strong feelings about these tools, and why many clients now ask whether AI was used.

Real-world examples

From a 2015 physics idea to a 2020 breakthrough

The idea began in a 2015 research paper, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics", by Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan and Surya Ganguli. Borrowing from physics, where diffusion describes how things spread out and mix, they proposed a model that slowly destroys the structure in data with noise and then learns to reverse that destruction. It worked, but the pictures were small and far behind the best image generators of the day, and the idea stayed in the background for five years.

In June 2020, Jonathan Ho, Ajay Jain and Pieter Abbeel at UC Berkeley published "Denoising Diffusion Probabilistic Models", often shortened to DDPM. By simplifying what the model is asked to learn, essentially the "guess the noise" task described above, they produced images that matched or beat the leading methods on standard tests. Researchers took notice. Within two years, diffusion had gone from a curiosity to the method nearly every major lab was using for images.

The lesson for creators is that the tools you use today rest on ordinary published science. Nothing about them is magic, and the basic recipe is public.

DALL·E 2 and Stable Diffusion put diffusion in everyone's hands (2022)

On 6 April 2022, OpenAI unveiled DALL·E 2. Its research paper, published a week later, describes a system that uses a diffusion model to turn an idea of an image into the finished picture. The results, often photorealistic and able to combine unrelated ideas in one picture, made headlines, but access was at first limited to a waitlist.

Four months later, on 22 August 2022, Stability AI announced the public release of Stable Diffusion, built on the "latent diffusion" research of Robin Rombach and colleagues at a German university group called CompVis. The difference was that anyone could download the model itself. The announcement said it would run on a consumer graphics card with 6.9 GB of video memory, and it came under a licence that allowed commercial use. Almost overnight, hobbyists, artists and developers were running it at home, building add-ons, and training their own variations. The release was influential enough that in March 2023 the programmer Simon Willison described chatbots running on laptops as language models "having their Stable Diffusion moment", a story told under parameters.

The prize-winning image that could not be copyrighted

On 29 August 2022, one week after Stable Diffusion's public release, a game designer from Colorado named Jason M. Allen won the blue ribbon in the digital arts category for emerging artists at the Colorado State Fair with "Théâtre D'opéra Spatial", an ornate, painterly scene. He had made it with Midjourney, an image generator from the same 2022 wave (Midjourney has not published full details of how its models work), then cleaned it up in Adobe Photoshop and enlarged it with an upscaling tool. The win set off a furious public argument about whether AI art was art.

The legal chapter matters more to working creators. Allen applied to register the copyright in September 2022. He told the U.S. Copyright Office that he had entered revisions and prompts at least 624 times to reach the first version of the image. The Office refused to register the work as submitted, and on 5 September 2023 its Review Board confirmed the refusal. It found that the Midjourney-generated image was not the product of human authorship and would have to be excluded from the claim; his own edits might be registrable, but Allen declined to disclaim the AI-made part. Allen sued in federal court in Colorado in September 2024. At the time of writing, that case was still waiting for a decision.

In January 2025, the Copyright Office published a wider report on AI and copyright. Its conclusion was that, with today's tools, prompts alone do not give a person enough control over the result to count as the author, although human-made parts of a work, and creative selection, arrangement or editing of AI output, can still be protected. If you sell AI images, read that report, and check the rules in your own country, which may differ.

Getty Images v. Stability AI

On 17 January 2023, Getty Images announced it had begun legal proceedings in London against Stability AI, saying the company had copied millions of its images without a licence to train Stable Diffusion. The case also turned on something users had noticed: some generated pictures contained a distorted version of the Getty Images watermark, a sign of how heavily those photos featured in what the model learned.

The High Court gave its judgment on 4 November 2025, and it went largely against Getty. Getty had dropped its main claim about the training itself, because there was no evidence the training had happened in the UK. On what remained, the judge held that the finished model does not store copies of the photos it learned from; it holds patterns learned during training. Getty won only a narrow point about watermarks appearing in a small number of outputs from some older versions of the model. In December 2025 the judge gave Getty permission to appeal the copyright question, and Getty has a separate case against Stability AI in California, refiled in August 2025. At the time of writing, neither had reached a final answer, so treat any confident claim that "training is legal" or "training is theft" with caution.

Sora: diffusion comes to video, and goes

On 15 February 2024, OpenAI previewed Sora, a model that could create video clips up to a minute long from a text prompt. OpenAI described it as combining a diffusion model with a transformer: it cuts video into small chunks across space and time and cleans them up together. The early samples also showed classic diffusion flaws, such as objects appearing and vanishing and people forgotten once they left the frame. Sora opened to paying ChatGPT subscribers on 9 December 2024. Then, in March 2026, OpenAI announced it was discontinuing Sora, and the Sora app closed on 26 April 2026. Other video tools, such as Runway, Veo and Kling, carry on. It's a reminder that the underlying technique can outlast any one product built on it.

An everyday scenario: a book cover mock-up

Imagine Dana, a self-published romance author, who needs a cover concept before hiring a designer. She types "a woman in a red coat on a rainy Paris street at night, cinematic, soft focus" and gets four options. She likes the third one's composition, so she notes its seed and reruns it with "snowy" instead of "rainy": same street, same pose, now in snow. Next she uploads a rough sketch of her own layout and uses image-to-image with a medium strength setting, so the tool keeps her title space at the top. She takes the result to her designer as a reference, not as the final cover, because she has read that the AI-made parts may not be protected by copyright and she wants a cover she fully owns.

Try it yourself

You need a web browser. Allow 20 minutes. These steps use a free public demo of FLUX.1 [schnell], a fast open image model, hosted on Hugging Face. At the time of writing it ran without an account, though busy periods and daily limits for visitors who aren't signed in can cause errors; if it fails, wait and try again, or use any image tool that shows you a seed and a steps setting. Check each tool's terms, including what you may do with the images, before using results commercially.

  1. Open the demo. Go to huggingface.co/spaces/black-forest-labs/FLUX.1-schnell and wait for the page to load.
  2. Make your first image. Type a short prompt in the box and press Run: “ a watercolor painting of a lighthouse at dawn “
  3. See the seed at work. Press Run again with the same prompt. You get a different lighthouse, because "Randomize seed" is ticked in Advanced Settings, so every run starts from different static.
  4. Lock the seed. Open Advanced Settings, untick "Randomize seed", and set Seed to 42. Run it twice. The two pictures should match, because the starting static is now the same.
  5. Change one word. Keep seed 42 and change "dawn" to "midnight". Most of the layout often survives while the lighting changes. This is the most useful habit in AI image work: fix the seed, change one thing, compare.
  6. Play with steps. Set "Number of inference steps" to 1, then 2, then 4, keeping seed 42. At 1 step you are seeing how far the model gets with a single clean-up pass. This particular model is trained for about 4 steps; many others use 20 to 50.
  7. Test the classic weak spots. Try a prompt with counting and text, then look closely: “ a hand holding exactly three red apples, and a shop sign that reads OPEN DAILY “ Count the apples and fingers, and check the spelling. Whatever you find, you have just tested the model rather than trusting it.
  8. Watch the noise disappear. Open the Diffusion Explainer listed under resources on this page. It shows how Stable Diffusion turns noise into a picture for real prompts, which makes the idea in "What it is" visible.

Common mistakes and misunderstandings

"It stitches together pieces of existing pictures." That's not how it works. A diffusion model doesn't keep a library of photos to cut and paste from; the UK High Court accepted in the Getty case that the model itself does not store the images it learned from. That doesn't mean it can never produce something very close to a picture it saw many times in training, especially famous artworks, logos or watermarks. Treat near-copies as a real risk, particularly when you prompt with an artist's name or a brand.

"More steps always means a better picture." Past a certain point, extra steps add waiting time and very little quality. Each tool has a sweet spot, and some are built to use only a few. If a picture looks wrong, changing the prompt or the seed usually helps more than adding steps.

"A longer, fancier prompt is always better." Piling on phrases like "8K, masterpiece, ultra detailed" was popular with early models and often does little with newer ones. Clear subject, setting, style and lighting usually beat a wall of keywords. Test it yourself with a fixed seed.

"If I made it, I own it." In the United States, the Copyright Office's position at the time of writing is that prompts alone don't make you the author of what the model produces, as Jason Allen found. Your own edits, drawings and arrangement can still count. Rules differ between countries, and each tool's terms say what you may do with the output. Check both before selling AI images or putting them on products.

"All AI image tools work the same way." Many use diffusion, but not all. OpenAI has described its image generation in ChatGPT since 2025 as a different, autoregressive approach. Tools also differ in what data they were trained on, what they let you do commercially, and how they handle real people and brands.

"The model understands what it's drawing." It has learned what things usually look like, not how they work. That's why you can still get a clock with 13 numbers or a bicycle with an impossible frame. Check anything that needs to be correct, such as text, hands, logos, maps and diagrams, before publishing. This is the image-world cousin of hallucination.

"The pictures are neutral." A model learns from whatever pictures it was fed, including the stereotypes in them. Ask for "a CEO" or "a nurse" without details and see who appears. If representation matters for your project, describe the people you want rather than leaving it to the model's defaults. See bias.

"It's too expensive to run myself." Some open models, including early versions of Stable Diffusion, run on a home computer with a decent graphics card. Big video models are another matter and still need data-centre hardware. Whether local use is worth the setup depends on how much control you want.

Related elements