Where Models Run: Cloud, Device, and Routing Between Models

Every earlier post in this part assumed a model on a server. Many products could run some of their model work on the user’s own device, and most could use more than one model. This post is about those decisions: where each request should run, and on which model.

Table of contents

The trade-offs

• Cost. In the cloud, every token is paid for, by the request or by the GPU-hour. On the device, inference is free to the provider: the user’s hardware and battery pay.
• Latency. A device skips the network round trip and any queue, but runs a small model on slow hardware. A data center adds tens to hundreds of milliseconds of network, then runs a large model on fast hardware. For short answers the device often wins; for long ones the server does.
• Privacy. Data that never leaves the device can’t leak from a server and needs no trust in its operator. For some data (health, messages, anything regulated) this decides the question outright.
• Quality. The largest models only run in data centers. A device model is a fraction of their size and capability.
• Availability. A device works offline; a server works on any device, including a ten-year-old phone.

What fits on a device

Memory decides most of it (Quantization from Scratch, Running LLMs on a Laptop). A phone has far less memory than a GPU server, shared with the operating system and every app. At 4 bits per weight, a 3B-parameter model is about 1.5 GB, a 7B model about 3.5 GB plus its KV cache, and anything much larger doesn’t fit. Decoding speed is bounded by memory bandwidth over weight size, the same arithmetic as on a GPU at a far smaller scale: a phone’s memory moves tens of GB per second, so a 1.5 GB model decodes at most a few tens of tokens per second, and pushing it that hard drains the battery and heats the phone.

Apple’s design shows the practical answer. Its 2025 on-device model has about 3 billion parameters, trained with quantization in the loop down to 2 bits per weight, and shares part of its KV cache across layers to save memory. Features that need more run on a much larger server model (a “parallel-track” mixture-of-experts) on Apple’s own data-center hardware (“Private Cloud Compute”) designed so that request data isn’t retained or accessible to Apple, by Apple’s account. Runtimes for running open models locally include llama.cpp (see Running LLMs on a Laptop), Apple’s MLX, and Google’s LiteRT; the lesson is the deployment decision, not the library.

Hybrid designs and routing

Most real systems are hybrids: a small model where it’s good enough, a large one where it isn’t. Deciding which, per request, is routing, and there are three common ways:

• A classifier router looks at the request alone and predicts which model should answer it, before any model runs. Cheap, but it must judge difficulty without seeing an attempt. RouteLLM (Ong et al., 2024) trained such routers (from a similarity-weighted ranking to a fine-tuned language-model classifier) on human preference data comparing strong and weak models’ answers, augmented with labeled and judge-scored data.
• A cascade tries the small model first and escalates when its answer looks unreliable: low confidence, a failed check, a refusal. FrugalGPT (Chen et al., 2023) built cascades over several API models with a learned scorer deciding whether to accept each answer. Every escalated request pays for both models, so a cascade pays off only when the small model handles most requests.
• A learned orchestrator chooses among several models, or combines them (Learned Orchestration’s Fugu).

Routing between models is the same idea as a mixture-of-experts router (Mixture of Experts) at a larger scale: a learned or heuristic function picks which expert handles each input. Inside a model, the experts are feed-forward blocks and the choice is per token; between models, the experts are whole models (or a device and a data center) and the choice is per request.

A cascade, measured

Two models stand in for a device model and a bigger served one: Qwen2.5-1.5B-Instruct (small) and Qwen2.5-3B-Instruct (large). Each answers 150 questions with checkable answers, of three kinds; the small model’s confidence is the geometric mean of the probabilities it gave its own answer’s tokens, leaving out the last one (normally the end-of-turn token); the cascade escalates every question below a threshold. The generation script, route.py, saves both models’ answers to answers.json, and prints a first summary. The timings come from an Apple M4 laptop, where the whole run takes about 2 minutes. Each model answers every question greedily, timed between two sync() calls, which wait for the machine’s GPU to finish:

@torch.no_grad()
def answer_all(path):
    tok = AutoTokenizer.from_pretrained(path)
    model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.float16).to(DEVICE).eval()
    out = []
    for q, truth, kind in items:
        ids = tok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True,
                                      return_tensors="pt", return_dict=True)["input_ids"].to(DEVICE)
        sync(); t0 = time.perf_counter()
        g = model.generate(ids, max_new_tokens=12, do_sample=False, temperature=None, top_p=None, top_k=None,
                           output_scores=True, return_dict_in_generate=True)
        sync(); dt = time.perf_counter() - t0
        new = g.sequences[0, ids.shape[1]:]
        logp = [torch.log_softmax(s[0].float(), -1)[t].item() for s, t in zip(g.scores, new)]
        text = tok.decode(new, skip_special_tokens=True).strip()
        right = re.sub(r"[^a-z0-9]", "", truth.lower()) in re.sub(r"[^a-z0-9]", "", text.lower())
        out.append({"right": right, "conf": float(np.exp(np.mean(logp[:max(1, len(logp) - 1)]))), "time": dt, "text": text})
    del model; empty_cache()
    return out

After both models have answered every question, the script tries each policy on the saved answers:

# The cascade: every question goes to the small model first; below the
# confidence threshold it is also sent to the large model, whose answer is used.
print(f"{'policy':34s} {'accuracy':>9s} {'sent to large':>14s} {'seconds/question':>17s}")
def report(name, use_large):
    acc = np.mean([L[i]["right"] if u else S[i]["right"] for i, u in enumerate(use_large)])
    cost = ts + np.mean(use_large) * tl if "cascade" in name or "oracle" in name else (tl if all(use_large) else ts)
    print(f"{name:34s} {acc:9.0%} {np.mean(use_large):14.0%} {cost:17.2f}")
report("always small", [False] * len(items))
report("always large", [True] * len(items))
for th in (0.5, 0.7, 0.8, 0.9, 0.95):
    report(f"cascade, escalate below {th:.2f}", [r["conf"] < th for r in S])
report("oracle cascade (escalate if wrong)", [not r["right"] for r in S])

Full script: route.py

The rest of route.py builds the 150 questions (50 capitals, 50 two-digit multiplications and 50 day-of-week sums, from a seeded generator), saves the answers to answers.json, and prints accuracy by kind and the small model’s median confidence on right and wrong answers.

150 questions; seconds per question: small 0.22, large 0.40

             small   large   (accuracy by kind)
capital          92%     88%
multiplication   54%     84%
weekday          18%     26%
all              55%     66%

policy                              accuracy  sent to large  seconds/question
always small                             55%             0%              0.22
always large                             66%           100%              0.40
cascade, escalate below 0.50             57%             7%              0.25
cascade, escalate below 0.70             60%            19%              0.29
cascade, escalate below 0.80             59%            30%              0.34
cascade, escalate below 0.90             63%            43%              0.39
cascade, escalate below 0.95             65%            55%              0.44
oracle cascade (escalate if wrong)       73%            45%              0.40

small model: confidence of right answers median 0.99, of wrong answers 0.78

That summary turned out to be wrong in two ways, which the second script fixes. Its answer matcher dropped accented letters, so “Brasília” failed, and it accepted one capital per country where there are documented alternatives (La Paz for Bolivia, Nur-Sultan for Kazakhstan’s capital in 2019–2022). Rescored, and with a look at which questions each policy escalates, with bootstrap intervals on the gain over the small model:

ALSO = {"Sucre": ["la paz"], "Astana": ["nur-sultan", "nursultan"], "Gitega": ["bujumbura"]}
fold = lambda t: re.sub(r"[^a-z0-9]", "", unicodedata.normalize("NFKD", t.lower()).encode("ascii", "ignore").decode())
def right(text, truth):
    return any(fold(a) in fold(text) for a in [truth] + ALSO.get(truth, []))
S = np.array([right(r["text"], it[1]) for r, it in zip(res["small"], items)])
L = np.array([right(r["text"], it[1]) for r, it in zip(res["large"], items)])
conf = np.array([r["conf"] for r in res["small"]])
kind = np.array([it[2] for it in items])
print(f"{'':16s} {'small':>6s} {'large':>6s}")
for k in ("capital", "multiplication", "weekday"):
    m = kind == k
    print(f"{k:16s} {S[m].mean():6.0%} {L[m].mean():6.0%}   small's confidence when wrong: median {np.median(conf[m & ~S]):.2f}")
print(f"{'all':16s} {S.mean():6.0%} {L.mean():6.0%}\n")

rng = np.random.default_rng(0)
def report(name, esc):
    acc = np.where(esc, L, S)
    boot = [np.where(esc, L, S)[i].mean() - S[i].mean() for i in (rng.integers(0, len(S), len(S)) for _ in range(2000))]
    lo, hi = np.percentile(boot, [2.5, 97.5])
    by = ", ".join(f"{k} {int((esc & (kind == k)).sum())}" for k in ("capital", "multiplication", "weekday"))
    print(f"{name:40s} {acc.mean():5.0%}  gain over small {acc.mean() - S.mean():+5.0%} [{lo:+.0%}, {hi:+.0%}]  escalated: {by}")
for th in (0.7, 0.9):
    report(f"cascade, escalate below {th}", conf < th)
report("route by type: only capitals to small", kind != "capital")
report("route by type: only multiplication to large", kind == "multiplication")
report("always large", np.ones(len(S), bool))
report("oracle (escalate exactly when small is wrong)", ~S)

Full script: rescore.py

Above this, rescore.py loads answers.json and rebuilds the same 150 questions by running the lines of route.py that make them.

                  small  large
capital             96%    94%   small's confidence when wrong: median 0.78
multiplication      54%    84%   small's confidence when wrong: median 0.86
weekday             18%    26%   small's confidence when wrong: median 0.67
all                 56%    68%

cascade, escalate below 0.7                61%  gain over small   +5% [+1%, +9%]  escalated: capital 1, multiplication 1, weekday 26
cascade, escalate below 0.9                65%  gain over small   +9% [+3%, +15%]  escalated: capital 6, multiplication 17, weekday 41
route by type: only capitals to small      69%  gain over small  +13% [+6%, +19%]  escalated: capital 0, multiplication 50, weekday 50
route by type: only multiplication to large   66%  gain over small  +10% [+5%, +16%]  escalated: capital 0, multiplication 50, weekday 0
always large                               68%  gain over small  +12% [+5%, +20%]  escalated: capital 50, multiplication 50, weekday 50
oracle (escalate exactly when small is wrong)   73%  gain over small  +17% [+11%, +23%]  escalated: capital 2, multiplication 23, weekday 41

Bigger isn’t better everywhere. The large model is far better at multiplication (84% against 54%), about the same at capitals (94% against 96%), and both are near chance at weekday arithmetic (26% and 18%, where a guess gets 14%). Overall: 56% small, 68% large, at 0.22 and 0.40 seconds a question.

The confidence cascade mostly escalated the wrong questions. At a threshold of 0.7 it sends 28 questions on, 26 of them weekday questions, where the large model is barely better than guessing; its gain, 5 points, is mostly those guesses landing right, with an interval running down to 1 point. The questions that needed the large model were the multiplications, and the small model was confidently wrong on them: a median confidence of 0.86 on its wrong products. Only at 0.9 does the cascade start catching them, by then escalating 64 of 150 questions. Overconfidence where the small model is weakest is the standard way cascades fail.

A trivial router does better. Send capitals to the small model and everything else to the large one: 69%, above the confidence cascade at any threshold and as good as always using the large model (68%) with a third fewer questions sent to it, using no confidence at all. The 69% is measured on the same questions the rule was chosen from, so the one-point edge over the large model is not evidence of a real difference. Routing by the kind of request is exactly what a classifier router learns, and here the kinds are easy to tell apart. The oracle row, 73%, shows what perfect routing between these two models would reach: higher than either alone, because each is right on some questions the other misses.

The economics here are deliberately mild: the large model is only 1.8 times slower than the small one. Between a local model and a frontier API the price ratio is often far larger, which makes every question the small model can safely keep worth far more, and makes a router that knows which questions those are worth building.

Evaluating a router

A router is a classifier, and is evaluated like one (Measuring a Classifier Honestly): what it should escalate, how often it does (recall), and how often what it escalates needed it (precision). Its cost–quality curve comes from sweeping the threshold; its thresholds, like any classifier’s, must be chosen on held-out data, not on the questions used to report it (this post’s 150 serve as both, which a real deployment shouldn’t copy); and the oracle row bounds what any router using the same two models could reach. Two practical cautions. Routers are trained and tuned on one mix of traffic, and the mix shifts. And the large model’s accuracy on escalated questions is lower than its average, because escalation selects the hard ones; measuring each model on all traffic overstates what routing will achieve.

Where this leads

• System Design for AI makes the same where-does-it-run decision for one latency-bound product: a hosted LLM’s 250 ms to first token, against tens of milliseconds for a self-hosted model with the conversation cached, is one line of a one-second budget.

Takeaways

• Device or cloud is a trade between cost, latency, privacy, quality and availability; memory decides what fits on a device.
• A 4-bit 3B model is about 1.5 GB; Apple ships a ~3B on-device model at 2-bit weights and escalates to a server model.
• Routing makes the decision per request: classifiers decide before any model runs, cascades after the small model tries, learned orchestrators choose or combine.
• Measured with a 1.5B and a 3B model on 150 questions: a cascade on the small model’s confidence escalated mostly weekday questions, where both models are near chance, and gained 5 points, because the small model was confidently wrong where the large one helped; routing by question type reached 69%, as good as always using the large model (68%) with a third fewer questions sent to it; perfect routing would reach 73%.
• Routing between models is mixture-of-experts at a larger scale, and a router is a classifier: measure it like one, on the traffic it will see.

References

Start with RouteLLM for learned routers and FrugalGPT for cascades, then Apple’s 2025 report for a hybrid device-and-server design.

• Chen, Zaharia and Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023): cascades over several API models, with a learned scorer that accepts or escalates each answer.
• Ong et al., RouteLLM: Learning to Route LLMs with Preference Data (2024): classifier routers trained on preference data between a strong and a weak model.
• Apple, Apple Intelligence Foundation Language Models: Tech Report 2025: the ~3B on-device model with 2-bit weights and KV-cache sharing, and the server model behind Private Cloud Compute.