Tag: cost
All the articles with the tag "cost".
From Function Calling to the Agent Loop
Updated:An agent is a model in a loop, with tools and a rule for stopping. What function calling is (a protocol around generated text: the model never runs anything), how a call is emitted and parsed, parallel calls, think-act-observe, stopping, and the transcript as append-only context, with its cost growing quadratically per turn unless cached. Then an agent searching this blog, on 1.5B and 3B models, measured against one retrieval and one answer: whether small models choose to use tools, what forcing a call fixes, and how they cite what they found.
Prompt Caching: Paying Once for a Prefix
Updated:API prompt caching is the KV cache and prefix sharing sold as a price. What is actually cached, why only an exact prefix can hit, breakpoints and lifetimes, the write and read prices and the break-even worked from published multipliers, the time a cached prefix saves measured on a 1.5B model, the gigabytes a provider holds for one, how to order a prompt, and how agents break their own cache.
Where Models Run: Cloud, Device, and Routing Between Models
Updated:Where a model runs is an optimization over cost, latency, privacy and quality, and routing turns it from one decision per product into one per query. The arithmetic of cloud against device; what fits in a phone's memory; hybrid designs like Apple's; routers, cascades and learned routing; how routing relates to mixture-of-experts; and a cascade measured between a 1.5B and a 3B model on 150 checkable questions, where routing by the kind of question beat routing on the small model's confidence.