Tag: expert-parallelism

All the articles with the tag "expert-parallelism".

• Mixture of Experts: More Parameters, Same FLOPs

  Updated:16 Oct, 2026

  A mixture of experts replaces a Transformer's MLP with many MLPs and a router that sends each token to a few of them, so a model can hold far more parameters than it uses per token. The router worked through, the load-balancing loss derived, DeepSeek-V3's bias balancing, capacity and dropped tokens, and shared and fine-grained experts. Then an MoE version of the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog, trained four ways, the parameter counts of Mixtral and DeepSeek-V3 recomputed from their configs, and what experts cost at serving time: memory, all-to-all traffic, and batches big enough to keep experts busy.