Only a few experts activate for each token, so the model can be very large while staying relatively fast and affordable to run.
17
Beginner
Mx
Mixture of Experts
Core Concepts
Mixture of Experts
A model split into specialist sub-networks.