Tag: routing
All the articles with the tag "routing".
Learned Orchestration: A Case Study of Sakana Fugu
Updated:What if the orchestrator of a multi-agent system were itself a model, trained to pick and instruct frontier models? Sakana's Fugu report (2026) answers with two designs: a router that reads one hidden state and picks one worker, trained by supervised fine-tuning and then evolution strategies, and a Conductor trained with GRPO to write whole multi-agent workflows in natural language. How each is built, the failure called orchestration collapse and its fix, and the reported results read critically: ahead of the best single model on 7 of 11 benchmarks each, against baselines gathered from providers and leaderboards rather than run under the same conditions.
Where Models Run: Cloud, Device, and Routing Between Models
Updated:Where a model runs is an optimization over cost, latency, privacy and quality, and routing turns it from one decision per product into one per query. The arithmetic of cloud against device; what fits in a phone's memory; hybrid designs like Apple's; routers, cascades and learned routing; how routing relates to mixture-of-experts; and a cascade measured between a 1.5B and a 3B model on 150 checkable questions, where routing by the kind of question beat routing on the small model's confidence.
Mixture of Experts: More Parameters, Same FLOPs
Updated:A mixture of experts replaces a Transformer's MLP with many MLPs and a router that sends each token to a few of them, so a model can hold far more parameters than it uses per token. The router worked through, the load-balancing loss derived, DeepSeek-V3's bias balancing, capacity and dropped tokens, and shared and fine-grained experts. Then an MoE version of the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog, trained four ways, the parameter counts of Mixtral and DeepSeek-V3 recomputed from their configs, and what experts cost at serving time: memory, all-to-all traffic, and batches big enough to keep experts busy.