Small and Specialized Models: When Not to Use a Frontier LLM

Part V: Modern architecture followed the largest models. Most calls to a model in production don’t need one. A support ticket needs a category, an email needs a yes or no on “is this spam”, a question needs a route, a form needs a field extracted. These are decisions: a small, fixed set of possible answers, made millions of times, where latency and cost matter and a confident wrong answer is worse than an honest “not sure”. This post is about the models that serve them: small language models, and models that don’t generate text at all.

Table of contents

What “small” buys

“Small” means three things that move together: fewer parameters, less memory, and lower latency. The link between the last two is direct. Generating one token reads every weight once (The KV Cache’s decode), and with one request at a time there’s nothing to amortize that read over, so generation speed is capped by memory bandwidth divided by the size of the weights. Worked for five sizes, in 16-bit weights and in 4-bit (Quantization from Scratch, later in the series, covers quantization):

# What "small" buys. Decoding one token reads every weight once, so with a
# batch of one, generation speed is bounded by memory bandwidth: tokens per
# second <= bandwidth / bytes of weights ("How a GPU Runs Deep Learning"'s roofline, memory-bound).
# Bandwidths: this laptop's GPU as measured in "How a GPU Runs Deep Learning" (93 GB/s), and an
# H100's 3.35 TB/s. Memory: the laptop's 16 GB, shared with everything else,
# and the H100's 80 GB; "-" marks weights that don't fit. Reading the KV
# cache ("The KV Cache") is left out: small at short context, not at long context.
LAPTOP, H100 = (93e9, 16e9), (3.35e12, 80e9)
models = [("0.5B", 0.5e9), ("1.5B", 1.5e9), ("3B", 3e9), ("8B", 8e9), ("70B", 70e9)]

def cap(device, size):
    bandwidth, memory = device
    return f"{bandwidth / size:5.0f}" if size < memory else "    -"

print(f"{'':6s} {'BF16':>8s} {'INT4':>8s}   tokens/s at most, batch 1:  laptop BF16 / INT4    H100 BF16 / INT4")
for name, n in models:
    bf16, int4 = 2 * n, 0.5 * n
    print(f"{name:6s} {bf16 / 1e9:6.1f}GB {int4 / 1e9:6.2f}GB   {'':26s}  {cap(LAPTOP, bf16)} / {cap(LAPTOP, int4)}    "
          f"{cap(H100, bf16)} / {cap(H100, int4)}")
# A check against a measurement: "Decoding" timed Qwen2.5-1.5B (1.54B
# parameters, 16-bit) at 41 ms per token on this laptop.
n = 1.54e9
print(f"\nQwen2.5-1.5B in 16 bits: {2 * n / 1e9:.2f} GB per token read; at the measured 41 ms per token that's "
      f"{2 * n / 0.041 / 1e9:.0f} GB/s, {2 * n / 0.041 / LAPTOP[0]:.0%} of the measured bandwidth")

Full script: bandwidth.py

           BF16     INT4   tokens/s at most, batch 1:  laptop BF16 / INT4    H100 BF16 / INT4
0.5B      1.0GB   0.25GB                                  93 /   372     3350 / 13400
1.5B      3.0GB   0.75GB                                  31 /   124     1117 /  4467
3B        6.0GB   1.50GB                                  16 /    62      558 /  2233
8B       16.0GB   4.00GB                                   - /    23      209 /   838
70B     140.0GB  35.00GB                                   - /     -        - /    96

Qwen2.5-1.5B in 16 bits: 3.08 GB per token read; at the measured 41 ms per token that's 75 GB/s, 81% of the measured bandwidth

The last line checks the arithmetic against a measurement: Decoding timed Qwen2.5-1.5B at 41 ms per token on this laptop, which is 75 GB/s of weights read, 81% of the bandwidth How a GPU Runs Deep Learning measured. Decoding one sequence really is a memory-bound job, and the table is a good guide. Two limits show up in it. Speed: a 3B model in 4 bits can reach about 60 tokens a second on this laptop, several times faster than people read, while a 3B model in 16 bits manages 16. Memory: an 8B model in 16 bits doesn’t fit in the laptop’s 16 GB, and a 70B model doesn’t fit in any form, nor in 16 bits on one 80 GB H100. The table also leaves out reading the KV cache (The KV Cache), which is small next to the weights at short context and isn’t at long context. Size matters most where the hardware is small or the volume is large: phones, laptops, and millions of calls a day.

How small models stay capable

A small model can’t hold as much as a large one, but three things close much of the gap on the tasks it’s meant for:

• Better data. Phi-3-mini, 3.8B parameters, trained on 3.3T tokens of “heavily filtered publicly available web data and synthetic data”, reports 69% on MMLU, a score that took far larger models a few years earlier, and is “small enough to be deployed on a phone” (Abdin et al., 2024).
• More data than compute-optimal. How LLM Architecture Evolved, 2017–2026 showed why small models are trained far past Chinchilla’s 20 tokens per parameter: for a fixed training budget, a smaller model trained on more tokens ends up a little worse than the compute-optimal one, but every token it later generates costs much less. Qwen2.5’s models, small and large, were trained on up to 18T tokens, over 10,000 tokens per parameter for the 1.5B.
• Distillation. Train the small model to match a large one’s full output distribution instead of only the next token. Gemma 2’s 2B and 9B models were trained “with knowledge distillation instead of next token prediction”; Llama 3.2’s 1B and 3B were pruned from Llama 3.1 8B and then trained on the 8B and 70B models’ logits. Knowledge Distillation derives why a teacher’s distribution carries more information per example than a label.

What small models give up is breadth: facts about the long tail of the world, and multi-step reasoning, which both scale with size. For a narrow task, they don’t need either.

System One and System Two

Kahneman described two modes of human thinking: System 1, fast, automatic, intuitive, and System 2, slow, deliberate, effortful (the labels are Stanovich and West’s). This post spells them out, System One and System Two, as the vendor below does. The labels fit two kinds of model call. Writing an essay, planning an agent’s steps, debugging code are System Two: open-ended, worth seconds of generation and a large model. Deciding which of four routes a question belongs to is System One: the answer is one of a few known options, and it should take milliseconds.

Using a generative language model for a System One call works, but it pays for things the task doesn’t use. It generates text that has to be parsed back into a decision. Its answer can be outside the set of valid options. And a generated label comes with no confidence, unless its token probabilities are read, and Measuring a Classifier Honestly showed those are only useful if they’re calibrated.

An experiment: routing the notebook’s questions

The Notebook Assistant has a System One decision in it from the start: the router of Features and Overfitting through Cross-Entropy and the Softmax Gradient, which sends each question to one of four routes, serving, architecture, agents or foundations. Its 200 labeled English questions make a direct test of three ways to make that decision. Each question gets one prediction from a model that never saw it, and every comparison comes with a 95% bootstrap interval over questions. On an Apple M4 laptop the whole experiment, routing.py, takes about 50 minutes. The language model is read two ways: typed, by the probability of every spelling of each route name, and free, by generating a few tokens and parsing them:

SPELLINGS = {r: [r, r.capitalize(), " " + r, " " + r.capitalize()] for r in ROUTES}
SPELLINGS["foundations"] += ["foundation", "Foundation", " foundation", " Foundation"]
SPELLINGS["agents"] += ["agent", "Agent", " agent", " Agent"]
cands = [(ROUTES.index(r), tok(s).input_ids) for r, forms in SPELLINGS.items() for s in forms]

def prompt(q, shots=None):
    if shots is None:
        text = (f"Classify the question into exactly one route.\nRoutes: {DESCRIBE}\nQuestion: {q}\n"
                "Answer with one word: serving, architecture, agents or foundations.")
    else:
        examples = "\n".join(f"Question: {r['question']}\nRoute: {r['route']}" for r in shots)
        text = (f"Classify the question into exactly one route.\nRoutes: {DESCRIBE}\n"
                f"Labeled examples:\n{examples}\n\nQuestion: {q}\n"
                "Answer with one word: serving, architecture, agents or foundations.")
    return tok.apply_chat_template([{"role": "user", "content": text}], add_generation_prompt=True,
                                   return_tensors="pt", return_dict=True)["input_ids"][0].tolist()

@torch.no_grad()
def typed(ids):
    """Probability of each route: the summed probability of every spelling's
    full token sequence after the prompt, scored in one batched forward."""
    L = max(len(c) for _, c in cands)
    seqs = [ids + c + [tok.eos_token_id] * (L - len(c)) for _, c in cands]
    h = m.model(torch.tensor(seqs, device=DEVICE)).last_hidden_state[:, len(ids) - 1:]   # only the scored positions
    logp = m.lm_head(h).float().log_softmax(-1)
    p = np.zeros(4)
    for i, (route, c) in enumerate(cands):
        p[route] += float(sum(logp[i, j, t] for j, t in enumerate(c)).exp())
    return p / p.sum(), p.sum()

def parse(text):
    """A forgiving parser: lowercase, drop punctuation and markdown, accept
    the singular and a leading prefix."""
    w = re.sub(r"[^a-z ]", " ", text.lower()).split()
    for word in w[:3]:
        for r in ROUTES:
            if word in (r, r.rstrip("s")) or (len(word) >= 5 and r.startswith(word)):
                return ROUTES.index(r)
    return None

Full script: routing.py

The rest of routing.py builds the 40 splits, trains the bag-of-words router and the MiniLM head on each, prompts Qwen zero-shot and with 20 examples drawn from the question’s own training split, times every method at batch 1, and scores them with bootstrap intervals. router_data.py, router4.py and softmax_basics.py are copies of the helpers from Features and Overfitting and Cross-Entropy and the Softmax Gradient.

Qwen2.5-1.5B zero-shot: prompts of 118 tokens on average; the route spellings carry 99% of the first-word probability (median); free answers unparseable: 0 of 200
Qwen2.5-1.5B, 20 examples: prompts of 439 tokens on average; the route spellings carry 98% of the first-word probability (median); free answers unparseable: 0 of 200

                                                 accuracy    ECE  batch-1 time
bag-of-words router                      0.655 [0.59, 0.71]  0.050        6.0 us
MiniLM + softmax head                    0.710 [0.65, 0.77]  0.035        2.6 ms
MiniLM, nearest centroid                 0.780 [0.72, 0.83]  0.158        2.6 ms
Qwen2.5-1.5B zero-shot, typed            0.540 [0.47, 0.61]  0.291      155.9 ms
Qwen2.5-1.5B zero-shot, free             0.525 [0.46, 0.59]    n/a      215.8 ms
Qwen2.5-1.5B, 20 examples, typed         0.565 [0.50, 0.64]  0.274      465.6 ms
Qwen2.5-1.5B, 20 examples, free          0.555 [0.49, 0.62]    n/a      521.4 ms

MiniLM head minus Qwen with examples (typed): +0.145 [+0.06, +0.23]
MiniLM head, answering only at confidence >= 0.6: 70% of questions answered, 82.7% of those right
MiniLM head, answering only at confidence >= 0.7: 57% of questions answered, 88.6% of those right
MiniLM head, answering only at confidence >= 0.8: 41% of questions answered, 92.7% of those right
Qwen with examples, typed, right by true route: serving 0.92, architecture 0.96, agents 0.12, foundations 0.26

Reading a language model’s answer is a measurement problem of its own. The typed readout sums the probability of each listed spelling of every route name, “foundations”, “Foundation”, ” foundation” and so on, and those spellings carry 98 to 99% of its probability for the first word, so almost nothing is lost. The free readout lowercases, strips punctuation and accepts the singular, and every one of its 400 answers parsed. A first version of this experiment read only one spelling per route and parsed strictly; it understated the language model by several points and overstated its miscalibration, which is a lesson in itself.

What the measurement shows:

• Accuracy. The bag-of-words router gets 66%, the sentence encoder with the same softmax head 71%, and the encoder with no trained head, each question sent to the route whose average training embedding it is nearest, 78%. The 1.5B language model gets 54% zero-shot and 57% with 20 labeled examples in its prompt. The encoder with a head beats the language model with examples by 14.5 points, $[+6, +23]$.
• Calibration. The trained softmax heads’ confidences mean what they say (top-label ECE, Measuring a Classifier Honestly, of 0.035 to 0.05). The language model’s typed probabilities are overconfident (0.27 to 0.29). Nearest-centroid has no trained confidence at all: its probabilities come from a temperature picked by hand, and its ECE of 0.16 measures that choice, not the method.
• An honest “not sure”. Because the encoder’s head is calibrated, its confidence can decide when to answer. Answering only when it is at least 70% sure, it answers 57% of questions and gets 88.6% of those right, and hands the rest to something slower: a larger model, or a person.
• Valid answers. Reading the probabilities of the route names makes every answer valid by construction, a typed decision; generating and parsing got the same accuracy within a point and a half, but only because the parser was written to forgive.
• Speed. One question at a time on this laptop, start to finish: 6 microseconds for the bag of words, 2.6 milliseconds for the encoder on the CPU, 156 milliseconds for the language model’s typed answer on the GPU (timed as one forward pass over the prompt, the least a typed readout needs), and 466 with the examples, whose prompt is four times longer. The encoder is 60 to 180 times faster than the language model; the bag of words, 26,000 times. The ratio belongs to the hardware: on the RTX 3090 rerun, routing.py still runs the encoder on the CPU, an x86 one there, and the encoder took 86.7 ms against 23.0 to 76.4 ms for the language model on the GPU, so the language model was the faster one.

Twenty examples barely help the language model (54% to 57%), and its errors lean hard: with examples it gets 96% of architecture questions and 92% of serving questions right, and 12% of agents questions and 26% of foundations ones. What this blog files under “agents” and “foundations” is a convention of this blog, and a few examples don’t teach it the way 120 labeled training questions do.

The lesson isn’t that language models are bad at classification: a larger model, or one fine-tuned on these 120 questions (Parameter-Efficient Fine-Tuning), would do better. It’s that for a fixed decision with labeled data, a small model trained on that data is the default to beat: more accurate here, honest about its uncertainty, valid by construction, and on the M4 two orders of magnitude faster. The simplest baseline, nearest centroid on good embeddings, was the most accurate of all.

Decision models: typed answers, not text

The router above is a decision model: it reads text and returns one of a fixed set of typed answers with a probability, and never generates text. In September 2026 TypeSafe AI announced a general-purpose model of this kind, Jev, which it calls a “System One model”. Everything in this section is vendor-reported unless stated otherwise.

As TypeSafe describes it, Jev takes text describing a program’s state plus a set of typed questions defined in advance (choices of up to 255 options, among others) and answers all of them in parallel, each with a probability. It is trained with a method TypeSafe calls RLCD, reinforcement learning for calibrated decisions, aimed at “epistemically honest probabilities” rather than human preference. TypeSafe reports end-to-end responses in 70 to 500 milliseconds against “3 to 329 seconds” for frontier models on such queries, input at $0.042 per million tokens, free output, and type errors that are impossible by construction. Its headline speed and cost ratios come from its own workflow evaluations, which it cautions are “on the higher end of real world gains”.

One independent study so far applies it. Rafe and Das (2026) used Jev to code about 196,000 Texas police crash narratives against a 27-question schema, reaching an F1 of 0.908 against human coders. On the same records they also ran two frontier language models: one scored 0.059 F1 higher than Jev, the other was statistically indistinguishable from it. Jev’s probabilities were not calibrated out of the box for this task: the authors recalibrated them, which cut calibration error by a factor of 3.3. And the case for Jev there is cost and throughput, argued from pricing rather than measured. It’s one outside application, but it is the kind of task the design targets: many typed questions about each of a very large number of texts, where a model slightly less accurate than the best frontier model can still be the right choice.

The design choices map onto this series. Typed outputs remove the parsing and invalid-answer problem the experiment above measured. Calibrated probabilities are Measuring a Classifier Honestly’s requirement, made the training objective, though, as the crash study shows, calibration on a new task still has to be checked and sometimes repaired. And skipping generation removes the sequential decoding loop of Decoding, which is where most of a language model’s latency goes.

Where each one fails

• A small language model fails at the long tail: rare facts, long multi-step reasoning, unfamiliar tasks. The usual fix is to route those calls to a larger model, which is what a router is for.
• A trained classifier fails when its labels or its inputs shift: a new route, a new kind of question, a new language (the Vietnamese questions of Measuring a Classifier Honestly). It needs labeled data and retraining, and it can only answer the question it was trained on.
• A general decision model answers questions it wasn’t trained on, but only questions with predefined answers. Anything that needs an explanation, a draft, or a plan is System Two work.
• A frontier model handles all of these, slowly and expensively, and, as measured here, isn’t automatically better at a narrow decision than a small model trained for it.

Where this leads

• Part VII: Training, fine-tuning and alignment trains models: fine-tuning (Fine-Tuning, and What It Destroys), cheap fine-tuning with LoRA (Parameter-Efficient Fine-Tuning), and distillation (Knowledge Distillation), the three ways a small model is made good at one thing.
• Quantization from Scratch quantizes, and Where Models Run comes back to routing between models of different sizes.

Takeaways

• Generating a token reads every weight once, so with one request at a time speed is capped by bandwidth over model size, and memory decides what fits at all: about 31 tokens a second at most for a 1.5B model in 16 bits on this laptop, which measured 41 ms per token, 81% of that cap.
• Small models stay capable through better data, training far past compute-optimal, and distillation from larger ones.
• Many production calls are decisions, not text. On this series’ router, a small encoder with a trained head beat a 1.5B language model given 20 examples, 71% against 57% ($+14.5$ points, $[+6, +23]$), calibrated against overconfident, and on the M4, 60 to 180 times faster; nearest centroid on the same embeddings reached 78%.
• Decision models return typed answers with probabilities and never generate; TypeSafe’s Jev is a general-purpose one, with vendor-reported claims and one independent application so far.
• Use the smallest thing that does the job, and measure it: accuracy, calibration and latency on the task’s own data.

References

Start with the Qwen2.5 report for the model family the experiment uses, then the Gemma 2 report for distillation into small models. Read the TypeSafe announcement next to Rafe and Das, which tests its claims on one outside task.

• Abdin et al. (Microsoft), Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024): a 3.8B model trained on heavily filtered and synthetic data.
• Gemma Team, Gemma 2: Improving Open Language Models at a Practical Size (2024): the 2B and 9B models trained by distillation.
• Meta, Llama 3.2: Revolutionizing edge AI and vision with open, customizable models (2024): 1B and 3B models pruned from 8B and distilled from 8B and 70B.
• Qwen Team, Qwen2.5 Technical Report (2024): the 0.5B to 72B family used here, and its 18T training tokens.
• Wang, Wei, Dong, Bao, Yang and Zhou, MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (NeurIPS, 2020): the distilled encoder behind all-MiniLM-L6-v2.
• Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP, 2019): the sentence encoders behind all-MiniLM.
• Kahneman, Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011): System 1 and System 2.
• Stanovich and West, Individual differences in reasoning: Implications for the rationality debate? (Behavioral and Brain Sciences, 2000): where the System 1 / System 2 labels come from.
• TypeSafe AI, Introducing System One Models & Jev (2026): the vendor announcement of Jev; its claims are vendor-reported.
• Rafe and Das, Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev) (2026): an independent application of Jev to crash narratives.