Aiodic

Back to the table

17 Beginner Mx Mixture of Experts Core Concepts

Mixture of Experts

A model split into specialist sub-networks.

What it is

A mixture of experts, usually shortened to MoE, is a way of building an AI model so that only part of it does the work for each piece of text. Instead of one giant block that is fully switched on for every word, the model contains many smaller sub-networks, called experts, plus a small traffic controller, called the router. For each token (a word or piece of a word), the router picks a few experts, those experts do the calculation, and the rest sit idle.

An everyday picture helps. Imagine a large repair workshop with sixty-four mechanics. A dense model, the traditional kind, works as if every mechanic has to look at every car that comes through the door, even a car that only needs a new wiper blade. A mixture-of-experts model has a receptionist at the front who glances at each job and sends it to two or three mechanics. The workshop still employs all sixty-four, and it still has to pay rent for the space they occupy, but each job only costs the time of a few of them. That is the whole trick: a very big workforce, and a small amount of work per job.

In a modern transformer, the design behind almost every chatbot, the experts usually replace one particular part of each layer, the "feed-forward" block, which is where much of the model's stored knowledge lives. The attention part, which lets words look at each other, is normally still shared by everyone. The router itself is a tiny neural network that gives each expert a score for the current token and sends the token to the highest scorers. Their outputs are blended, weighted by those scores, and passed on to the next layer, where a different router makes a fresh choice.

This leads to the idea you'll see in every MoE announcement: two different parameter counts. The total parameters are every learned number in the model, all the experts included. The active parameters are the ones actually used for a single token. Mistral's Mixtral 8x7B, for example, has 46.7 billion parameters in total but uses 12.9 billion per token. The total decides how much memory the model needs. The active count decides roughly how much computing each word takes, which is what drives speed and cost.

Where the idea came from

MoE is older than most people assume. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in the journal Neural Computation. They described a system of several separate networks, each of which learns to handle a subset of the training cases, with a gating network deciding which one to trust. Their test task was telling vowel sounds apart, and the system learned to divide the problem into simpler pieces on its own.

The idea sat mostly in research for a quarter of a century. Then in January 2017, a Google team including Noam Shazeer, Geoffrey Hinton and Jeff Dean published "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer". They put up to thousands of experts inside one layer, reached 137 billion parameters, and reported more than a thousandfold increase in model capacity with only minor losses in computational efficiency. The word "sparsely" is the key: only a sparse handful of experts is switched on at a time.

In January 2021, William Fedus, Barret Zoph and Noam Shazeer at Google published the Switch Transformer, which simplified things further by sending each token to just one expert. They trained models of up to a trillion parameters and reported up to seven times faster pretraining than a comparable standard model using the same computing resources. From there, the design moved from research papers into the products people use every day.

Why it matters

It breaks the link between size and cost. For years the rule in AI was simple: a bigger model knows more and does more, but every extra parameter makes every word more expensive to produce. MoE loosens that rule. A company can build a model with the knowledge capacity of a huge network while paying, per word, closer to the price of a much smaller one. That's why so many of the most capable models released since 2024 use it.

It explains why AI got cheaper so fast. When you notice that a new model is both better and cheaper than last year's, part of the reason is often MoE. Serving a model to millions of people is dominated by the cost of inference, the computing done every time the model produces an answer. Cutting the work per token by a factor of five or ten flows straight into prices, speed, and how many users one data centre can handle.

It's become the default at the frontier. The makers of several major model families say publicly that they use it. Google said in February 2024 that Gemini 1.5 uses a mixture-of-experts architecture. Meta called Llama 4, released in April 2025, "our first models that use a mixture of experts (MoE) architecture". Alibaba's Qwen3 family, released at the end of April 2025, includes MoE models, and OpenAI's open-weight gpt-oss models, released in August 2025, are MoE too. Some companies don't disclose their designs at all, so you often can't tell. But knowing the term lets you read model announcements without getting lost.

It changes what the numbers on a model card mean. Before MoE, "70B" told you almost everything about a model's size, speed and memory needs. Now you need both numbers. A model advertised as having 400 billion parameters might use only 17 billion per word, and that difference matters if you're choosing a model to run on your own computer or comparing prices between providers.

It has real trade-offs, which is why not everyone uses it. Every expert has to sit in memory, ready in case the router picks it, so an MoE model needs far more memory than its per-word cost suggests. Training is also trickier. If the router falls into the habit of favouring a few popular experts, those get overloaded while others barely learn anything, so builders add balancing rules to spread the work. And small models, the kind that run on phones and laptops, are still often dense, because on a device with limited memory, storing idle experts is a poor use of space.

Real-world examples

Mixtral 8x7B: the open model that made MoE famous

On 11 December 2023, the Paris start-up Mistral announced Mixtral 8x7B and released its weights, the file of learned numbers, under the permissive Apache 2.0 licence, so anyone could download and use it. Each layer has eight groups of experts, and for every token a router network picks two of them and adds their outputs together. The result: 46.7 billion parameters in total, 12.9 billion used per token.

Mistral said the model matched or beat GPT-3.5 on most standard benchmarks and ran inference six times faster than Meta's Llama 2 70B while outperforming it on most benchmarks. For many developers it was the first time they could download a genuinely strong MoE model and try it themselves, and it turned "mixture of experts" from a research phrase into something people discussed on forums and in newsletters.

The paper Mistral published in January 2024 contains the most useful lesson in this article. The researchers wanted to know whether the experts had specialised by subject, one for maths, one for biology, one for philosophy. They measured which experts were chosen for text from different fields and wrote that they were surprised to find no obvious patterns based on topic. What they did find was structure of a different kind: the Python word "self" tended to go to the same expert, indentation in code was consistently sent to the same experts, and consecutive tokens were often routed together. The experts had divided the work along lines that made sense to the model, not along the subject headings a person would choose.

DeepSeek-V3: hundreds of experts and a price tag that shook the market

In late December 2024, the Chinese lab DeepSeek published the technical report for DeepSeek-V3, a mixture-of-experts model with 671 billion parameters in total and 37 billion active per token. Its design pushed the idea much further than Mixtral: each MoE layer has 256 "routed" experts, of which the router picks 8 for each token, plus one "shared" expert that every token always passes through, to hold common knowledge so the specialists don't each have to relearn it. DeepSeek also described a new way to keep the experts evenly loaded without the extra penalty rule most earlier MoE models used during training.

The report's cost figure travelled much further than its diagrams. DeepSeek said the full training run took 2.788 million hours on Nvidia H800 chips and that, assuming a rental price of 2 dollars per chip-hour, this came to 5.576 million dollars. The report itself notes that the figure covers only the final official training run, not the earlier research and experiments. That caveat was often lost in the retelling.

In January 2025, DeepSeek's free assistant app, built on its newer R1 model, which used V3 as its foundation, climbed the app store charts. On 27 January 2025, Nvidia's shares fell about 17 percent in one day, and Reuters reported a loss of 593 billion dollars in market value, the largest one-day loss for any company on Wall Street at the time. Many things drove that reaction, not MoE alone, but the design was a central part of how DeepSeek got so much capability from so little computing per token. For the wider public, it was the moment the efficiency argument for MoE became front-page news.

Llama 4: a major lab switches over

Meta's first three generations of Llama models were dense. On 5 April 2025, Meta announced Llama 4 and described it as its first MoE family. Llama 4 Scout has 109 billion parameters in total across 16 experts, and Llama 4 Maverick has 400 billion in total with 128 routed experts plus a shared expert; both use 17 billion active parameters per token. Meta explained the choice in the same terms as everyone else: a single token activates only a fraction of the total parameters, which makes the model cheaper to train and serve. When one of the best-known open model families changes its basic design, it's a strong sign the industry has changed its mind.

An everyday scenario: choosing a model to run at home

Imagine Tomas, a freelance editor who wants to run an AI model on his own computer so client manuscripts never leave his desk. He finds two open models that score similarly on the tests he cares about. One is a dense model with 32 billion parameters. The other is an MoE model advertised as "30B-A3B", meaning 30 billion in total and 3 billion active.

He expects them to behave alike, since the headline sizes are nearly the same. They don't. Both need roughly the same amount of memory, because every expert in the MoE model has to be loaded even though most sit idle for any given word. But the MoE model writes several times faster, because each word only passes through a tenth of its parameters. For summarising long chapters, that speed is the difference between waiting and not waiting. Tomas picks the MoE model, and the only thing he had to understand to make that choice was the difference between the two numbers.

Try it yourself

You don't need to install anything. You need a web browser and, for the last step, any AI chatbot. Allow twenty minutes.

  1. Decode a model name. Go to huggingface.co, the website where most open models are published, and search for "Qwen3-30B-A3B". The "A3B" in the name means about 3 billion active parameters, and the "30B" is the total. Now search for "Mixtral-8x7B". Write down the two numbers you'd expect for each, then check them against the model's description page or the figures in this article.
  2. Look inside a real MoE model. On the Mixtral-8x7B-v0.1 page, open the "Files and versions" tab and click config.json. This is the model's settings file. Look for num_local_experts (the number of experts per layer, 8) and num_experts_per_tok (how many the router picks for each token, 2). Then open the same file for "deepseek-ai/DeepSeek-V3" and find n_routed_experts, n_shared_experts and num_experts_per_tok. You've just read the core design of two famous models straight from their makers' files.
  3. Work out the fraction. For each model, divide the active parameters by the total. Mixtral uses about 28 percent of its parameters per token. DeepSeek-V3 uses about 6 percent. That percentage is a rough guide to how much work the model saves compared with a dense model of the same total size.
  4. Check what a model maker actually says. Pick a model you use, such as the one behind your favourite chatbot, and search for its official announcement or model card. Look for the words "mixture of experts", "MoE", "active parameters" or "sparse". If the maker doesn't say, write "not disclosed". Articles and forum posts often claim a model is MoE based on rumour; only the maker's own statement settles it.
  5. Ask a chatbot to test your understanding. Paste this prompt into any chatbot:
I'm learning about mixture-of-experts AI models. Ask me five short
questions, one at a time, to check whether I understand the difference
between total and active parameters, what the router does, and why
MoE models need a lot of memory. After each answer, tell me what I
got right and correct anything I got wrong in plain language.
  1. Try the "experts" myth on the chatbot. Ask it: "In a mixture-of-experts model, is there a maths expert and a French expert?" A good answer will say no, or not in that human sense. If it confidently says yes, you've found a useful example of why you check claims against sources such as the Mixtral paper.

Model details, names and settings files change as new versions come out, so treat the numbers you find as correct at the time of writing and check the maker's current page.

Common mistakes and misunderstandings

Thinking the experts are specialists in human subjects. This is the biggest one, and the name encourages it. It's natural to picture a maths expert, a French expert and a poetry expert, with the router acting like a receptionist who reads the question and calls the right department. That's not what happens. Nobody assigns topics to the experts. The router and the experts are trained together, and the model finds whatever split of the work makes its predictions better. When Mistral checked, it found no obvious topic pattern; what it found instead was routing by things like code indentation and particular words. Other research, summarised in Hugging Face's guide, found experts that handle groups of tokens such as punctuation or proper nouns. The splits are real, but they are the model's own, and often not something a person would recognise as a subject.

Thinking the router picks experts once per question. Routing happens for every token, in every MoE layer, separately. A single sentence can pass through dozens of different combinations of experts on its way through the model. There is no moment where the whole question is handed to "the right expert".

Assuming MoE models are small because they're cheap to run. They save computing, not memory. Every expert must be loaded and ready, because the router might need any of them for the next token. Hugging Face's guide points out that Mixtral needs enough memory to hold about 47 billion parameters, even though each token only uses about 13 billion. If you plan to run a model yourself, the total count is the one that tells you whether it will fit.

Reading "8x7B" as eight times seven. Mixtral 8x7B has 46.7 billion parameters, not 56 billion, because only the expert blocks are multiplied by eight. The attention layers and other parts are shared by all the experts. Model names are shorthand, not arithmetic.

Comparing an MoE total with a dense model's size. A 400-billion-parameter MoE model and a 400-billion-parameter dense model are very different things. The dense model uses all 400 billion for every word and is far more expensive to run. Compare active with dense, and total with total, and look at real results before assuming either is better.

Believing every big model is MoE. It's common to read that a particular chatbot "is definitely a mixture of experts". Unless the company has said so, that's a guess. OpenAI, for example, did not publish GPT-4's architecture. Only rely on the maker's own announcement, technical report, or model card.

Treating a headline training cost as the whole cost. DeepSeek's 5.576 million dollar figure was carefully labelled in its own report as covering only the final training run, and excluding earlier research and experiments. MoE genuinely made training cheaper, but "a frontier model for six million dollars" was never what the report claimed.

Assuming MoE is always better. It isn't. MoE models are harder to train, can be harder to fine-tune, and waste memory on small devices. For a model that has to run on a phone, a well-trained dense model is often the better choice. MoE is a trade: more memory in exchange for less computing per word.

Related elements