Many expert sub-networks, but a router sends each token to only a few of them.
- RouterScores which experts suit this token.
- ExpertsFeed-forward blocks that specialise.
- Top-kOnly the best 2 of 8 experts run.
- CombineBlend the chosen experts' outputs.
Explain 1/5
Source: Hugging Face: Mixture of Experts explained