Tag: deployment
All the articles with the tag "deployment".
Where Models Run: Cloud, Device, and Routing Between Models
Updated:Where a model runs is an optimization over cost, latency, privacy and quality, and routing turns it from one decision per product into one per query. The arithmetic of cloud against device; what fits in a phone's memory; hybrid designs like Apple's; routers, cascades and learned routing; how routing relates to mixture-of-experts; and a cascade measured between a 1.5B and a 3B model on 150 checkable questions, where routing by the kind of question beat routing on the small model's confidence.