Features and Overfitting: How a Model Memorizes, and How L1 and L2 Stop It

Learning Is Optimization ended with a router that got 19 of 24 reader questions right. It used one feature: a count of words like “cache”, “batch” and “latency”. Its five mistakes all had the same cause. The feature couldn’t see what the question was about, only whether it used a particular vocabulary.

The obvious fix is to stop hand-picking the vocabulary and give the model every word as a feature, letting it learn which ones matter. This post makes that change, and it immediately runs into the central problem of machine learning. The richer model fits its training questions perfectly and does worse on new ones. We’ll see exactly why, watch the weights run away to infinity, and derive the two standard penalties that stop them. L2 shrinks every weight. L1 sets most of them to exactly zero. Then we’ll find that on this data, one well-chosen feature matches all of them, and learn how to tell which differences are real when every comparison reuses the same 200 questions. This post is half of a pair. Its sibling, Measuring a Classifier Honestly, asks the question this post only uses accuracy and loss for: how to measure a classifier honestly.

Table of contents

From text to features

A model can only multiply numbers, so every input has to become a vector first. The choices made in that step, called feature engineering, decide what the model is able to learn before training starts. A linear model can only weigh the features it’s given.

For text, the standard first representation is a bag of words:

1. Split each question into tokens: here, lowercase runs of letters and digits.
2. Build a vocabulary from the training questions: every distinct token gets a column.
3. Represent a question as a 0/1 vector, with a 1 in the column of each word it contains.

“Does vLLM use paged attention?” becomes a vector with ones in the columns for does, vllm, use, paged and attention, and zeros everywhere else. Each word is a category with its own dimension, as in one-hot encoding. A question is the one-hot vectors of its words added together and clipped to 1, a multi-hot vector. Word order is thrown away, which is where the name comes from.

A few decisions hide in those three steps, each of which matters:

• Presence or counts. We use 0/1 presence. Counts matter for long documents, but a word rarely repeats in a one-sentence question.
• The vocabulary comes from training data only. A word that first appears in a new question has no column, so it’s silently dropped. That’s correct, since the model has no weight for it, and it has a large consequence for the Vietnamese questions in Measuring a Classifier Honestly.
• What counts as a token. Our rule splits “TensorRT-LLM” into tensorrt and llm, and treats “token” and “tokens” as unrelated words. Better tokenization (stemming, subwords) is itself feature engineering. Byte-Pair Encoding covers how language models do it.
• Weighting. Classic retrieval systems down-weight words that appear everywhere, like the or how, with TF-IDF. We keep plain 0/1 and let the model learn weights, which turns out to have a cost.

A real dataset

The router now needs more than 24 questions. This post uses 200 English reader questions, written by hand for this series and labeled with the part of the blog that should answer them: 50 each for serving, architecture, agents and foundations. The 24 questions of Learning Is Optimization are included. There are also 20 Vietnamese questions, held back for Measuring a Classifier Honestly. The file is router-questions.csv, and every result below comes from it.

For now the router stays binary: serving versus everything else. So a quarter of the questions are positive. A router that always says “not serving” is right 75% of the time, a baseline worth keeping in mind.

The helper router_data.py reads the CSV, keeps the English questions, and lowercases each question into words with tokens() (runs of lowercase letters and digits, so punctuation is dropped). Its two functions split the questions and turn them into features:

def split(seed):
    """30 train / 10 validation / 10 test questions per route."""
    rng = np.random.default_rng(seed)
    train, val, test = [], [], []
    for route in ROUTES:
        group = [r for r in english if r["route"] == route]
        for k, i in enumerate(rng.permutation(len(group))):
            (train if k < 30 else val if k < 40 else test).append(group[i])
    return train, val, test

def featurize(train, *others):
    """Bag of words: one 0/1 column per word seen in *training* questions."""
    vocab = sorted({t for r in train for t in tokens(r["question"])})
    column = {t: j for j, t in enumerate(vocab)}
    out = []
    for rs in (train, *others):
        X = np.zeros((len(rs), len(vocab)))
        for i, r in enumerate(rs):
            for t in set(tokens(r["question"])):
                if t in column:          # words never seen in training are dropped
                    X[i, column[t]] = 1
        y = np.array([r["route"] == "serving" for r in rs], dtype=float)
        out.append((X, y))
    return out, vocab

Full script: router_data.py

Every script below imports it.

Three sets, three jobs

Once the model can fit its training data perfectly, as this one will, the training score says nothing about how it will do on questions it hasn’t seen. So the data is split three ways, and each part has one job.

• The training set (120 questions, 30 per route) is what the gradient sees.
• The validation set (40 questions) is for decisions about training: how long to train, how strong a penalty to use, which features to keep.
• The test set (40 questions) is looked at once, at the end, to report how the final model does. If it influences any decision, it stops being a test.

The split is stratified: each route contributes the same share to every set. That keeps a random split from putting, say, most of the serving questions in the test set.

With only 40 validation questions, one question moves accuracy by 2.5 points, so a single split is noisy. Where it matters, results below are averaged over 20 random splits. This is repeated hold-out validation, a close cousin of $k$-fold cross-validation, which rotates every example through the validation role once. Both answer the same question: how much of a difference is signal, and how much is which questions happened to land in which set?

Averaging over splits comes with a trap, and we’ll fall into it on purpose before climbing out. The 20 splits reuse the same 200 questions, so their results aren’t independent. A standard error computed as if they were is too small. The section How sure are we? measures by how much.

Overfitting, watched

Train the bag-of-words router with no penalty and record both losses as it goes. This is the same logistic regression as in Learning Is Optimization, with 379 weights instead of one:

sigmoid = lambda z: 1 / (1 + np.exp(-z))

def log_loss(p, y):
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p)).mean()

def train(X, y, lam=0.0, penalty="l2", eta=0.5, steps=4000):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(steps):
        p = sigmoid(X @ w + b)
        grad_w, grad_b = X.T @ (p - y) / len(y), (p - y).mean()
        if penalty == "l2":                   # weight decay
            w -= eta * (grad_w + lam * w)
        else:                                 # L1: step, then soft-threshold
            w -= eta * grad_w
            w = np.sign(w) * np.maximum(np.abs(w) - eta * lam, 0.0)
        b -= eta * grad_b
    return w, b

if __name__ == "__main__":
    # 1. One split, no penalty: watch training and validation loss diverge.
    (train_set, val_set, _), vocab = featurize(*split(seed=0))
    (Xtr, ytr), (Xva, yva) = train_set, val_set
    print(f"{len(vocab)} words, {len(ytr)} training questions")
    w, b = np.zeros(Xtr.shape[1]), 0.0
    for step in range(1, 25601):
        p = sigmoid(Xtr @ w + b)
        w -= 0.5 * Xtr.T @ (p - ytr) / len(ytr)
        b -= 0.5 * (p - ytr).mean()
        if step in (100, 400, 1600, 3200, 6400, 25600):
            print(f"step {step:5d}: train loss {log_loss(sigmoid(Xtr @ w + b), ytr):.4f}"
                  f"  val loss {log_loss(sigmoid(Xva @ w + b), yva):.4f}"
                  f"  |w| {np.linalg.norm(w):5.2f}")

Full script: overfit.py

379 words, 120 training questions
step   100: train loss 0.2138  val loss 0.3951  |w|  4.02
step   400: train loss 0.0701  val loss 0.3196  |w|  8.32
step  1600: train loss 0.0175  val loss 0.2880  |w| 13.43
step  3200: train loss 0.0086  val loss 0.2870  |w| 16.01
step  6400: train loss 0.0043  val loss 0.2918  |w| 18.56
step 25600: train loss 0.0011  val loss 0.3117  |w| 23.64

Three columns, three stories:

• Training loss heads to zero. By step 25,600 the model assigns its training labels a probability of about 0.999 (the geometric mean, $e^{-0.0011}$).
• Validation loss falls until around step 2,500, then rises. From that point on, every extra step makes the model better at its 120 training questions and worse at new ones. That turning point is overfitting, measured.
• The weight norm keeps growing, and it never stops.

The last column is the real diagnosis. It’s worth understanding why it happens, because it’s also the key to the fix.

Why the weights run to infinity

There are 379 features and only 120 questions. With more dimensions than points, there is often a hyperplane that separates the positives from the negatives perfectly: some $w$ with $w^\top x_i + b > 0$ for every serving question and $< 0$ for every other. Here there is: with no penalty, every one of the 20 splits below reaches 100% training accuracy. The training data is linearly separable. Now take that separating $w$ and scale it by $c > 1$. Every logit $z_i$ gets $c$ times further from zero, on the correct side, so every term of the log-loss shrinks:

So the loss has no minimum at any finite $w$. It only approaches zero as $\|w\| \to \infty$, and gradient descent obligingly walks off in that direction forever. In the table, the norm grows by about 2.6 each time the step count doubles, slowing roughly like $\log t$, which is what theory predicts for gradient descent on separable data.

Large weights are how this model overfits. A weight of 5 on a word that happened to appear in two training questions means that word alone moves the log-odds by 5, turning a probability of 0.5 into 0.99. The model becomes certain about patterns it saw twice. On new questions, some of those certainties are wrong, and log-loss punishes confident mistakes hardest. That’s the rising validation curve.

The first fix is already visible in the table: stop early, around step 2,500, where validation loss bottoms out. Early stopping works, and it’s standard when fine-tuning large models. (Pretraining is different: it usually passes over the data about once, so it rarely gets the chance to memorize.) But early stopping controls the weights only indirectly, through the step count. A penalty controls them directly.

L2: shrink every weight

Add the size of the weights to the loss:

The penalty’s gradient is $\lambda w$, so the update becomes

Here $\lambda$ is the penalty strength, not the eigenvalue it stood for in Learning Is Optimization. Every step first shrinks every weight by the factor $(1 - \eta\lambda)$, then takes the usual gradient step. This is why L2 regularization is also called weight decay. With $\eta = 0.5$ and $\lambda = 0.003$, each weight loses 0.15% of itself per step unless the data keeps pushing it back. On separable data, the pull toward infinity now meets a pull back toward zero, and they balance at a finite $w$. The loss has a true minimum again. There’s a second, quieter benefit that connects to Learning Is Optimization. The penalty adds $\lambda I$ to the Hessian, so it raises every curvature eigenvalue by $\lambda$. The flattest directions of the valley are lifted most in relative terms, which lowers the condition number and makes gradient descent converge faster.

The bias $b$ is left out of the penalty on purpose. It sets the base rate (here, “a quarter of questions are serving”) and shrinking it toward zero would bias every prediction toward 50%.

Where the penalty comes from

L2 isn’t an arbitrary add-on. It follows from a belief about the weights before seeing data. Suppose each weight is drawn from a Gaussian prior centered at zero, $w_j \sim \mathcal{N}(0, \sigma^2)$. Choosing the most probable weights given both the data and the prior is maximum a posteriori (MAP) estimation:

The first term is the log-loss, summed over examples. The second is the L2 penalty. Dividing through by $n$ to use the mean loss gives $\lambda = 1/(n\sigma^2)$. A strong penalty is a confident belief that weights are small. As $n$ grows, the data term dominates and the prior matters less, which is why regularization matters most exactly when data is scarce, as it is here.

L1: set most weights to zero

Replace the square with an absolute value:

This looks like a small change and behaves completely differently. The L2 penalty’s pull, $\lambda w_j$, fades as a weight gets small, so weights shrink toward zero but never reach it. The L1 penalty’s pull is $\lambda$ times the sign of $w_j$: the same size whether the weight is 5 or 0.001. So it can push a weight all the way to zero and hold it there.

Gradient descent can’t use that gradient directly, because $|w_j|$ has no derivative at zero, which is exactly where we want weights to land. The clean way is to split each step in two. Take the usual gradient step on the log-loss, getting a candidate $v_j$. Then solve the one-dimensional problem “stay close to $v_j$, but pay $\lambda$ per unit of size”:

For $u > 0$ the derivative is $(u - v_j)/\eta + \lambda$, which is zero at $u = v_j - \eta\lambda$. That’s only valid if $v_j > \eta\lambda$, and the case $u < 0$ is the mirror image. Otherwise the minimum is at $u = 0$. So:

This is the soft-threshold (the step is a proximal gradient step). With $\eta = 0.5$ and $\lambda = 0.008$, the threshold is 0.004 per step. A candidate weight of 0.30 becomes 0.296. A candidate of 0.003 becomes exactly 0. A weight at zero stays there as long as the size of its gradient is below $\lambda$, so the pull of the data can’t beat the pull of the penalty. The else branch of train() in overfit.py is this line.

The difference is easiest to see with two weights. Suppose the data alone would put them at $v = (2,\ 0.3)$, and the loss near there is the round bowl $\frac{1}{2}\|w - v\|^2$. Add each penalty with strength $0.5$:

• L1 applies the soft-threshold to each coordinate: $(2 - 0.5,\ 0) = (1.5,\ 0)$. The small weight is not shrunk. It is removed.
• L2 scales the whole vector: $w = v/(1 + 0.5) = (1.333,\ 0.2)$. Both weights shrink by the same factor, and neither reaches zero, however large $\lambda$ gets.

The prior behind L1 is the Laplace distribution, $p(w_j) \propto e^{-|w_j|/s}$, which has a sharp peak at zero. It encodes the belief that most weights are zero and a few are not. Geometrically, the set of weights with $\|w\|_1 \le t$ is a diamond with corners on the axes, and the loss’s contours usually first touch a diamond at a corner, where some coordinates are exactly zero. The L2 ball is round, with no corners to prefer.

The sweep

Now run both penalties at several strengths, each averaged over 20 splits. sweep.py imports train() and the losses from overfit.py:

# 2. Average every setting over 20 random splits: one 40-question
#    validation set is too small to trust on its own.
def evaluate(lam, penalty, seeds=range(20)):
    tr_acc, va_loss, va_acc, nonzero = [], [], [], []
    for seed in seeds:
        ((Xtr, ytr), (Xva, yva), _), _ = featurize(*split(seed))
        w, b = train(Xtr, ytr, lam, penalty)
        p = sigmoid(Xva @ w + b)
        tr_acc.append(((sigmoid(Xtr @ w + b) >= 0.5) == ytr).mean())
        va_loss.append(log_loss(p, yva))
        va_acc.append(((p >= 0.5) == yva).mean())
        nonzero.append((np.abs(w) > 0).sum())
    se = np.std(va_loss) / np.sqrt(len(va_loss))
    return (f"{penalty} λ={lam:<6}: train acc {np.mean(tr_acc):.3f}  "
            f"val loss {np.mean(va_loss):.3f} ± {se:.3f}  "
            f"val acc {np.mean(va_acc):.3f}  nonzero weights {np.mean(nonzero):.0f}")

for lam in [0.0, 0.001, 0.003, 0.01, 0.03, 0.1]:
    print(evaluate(lam, "l2"))
for lam in [0.001, 0.004, 0.008, 0.016]:
    print(evaluate(lam, "l1"))

Full script: sweep.py

l2 λ=0.0   : train acc 1.000  val loss 0.480 ± 0.034  val acc 0.818  nonzero weights 383
l2 λ=0.001 : train acc 1.000  val loss 0.421 ± 0.021  val acc 0.804  nonzero weights 383
l2 λ=0.003 : train acc 1.000  val loss 0.416 ± 0.015  val acc 0.794  nonzero weights 383
l2 λ=0.01  : train acc 0.987  val loss 0.435 ± 0.010  val acc 0.784  nonzero weights 383
l2 λ=0.03  : train acc 0.887  val loss 0.473 ± 0.006  val acc 0.753  nonzero weights 383
l2 λ=0.1   : train acc 0.751  val loss 0.516 ± 0.002  val acc 0.750  nonzero weights 383
l1 λ=0.001 : train acc 1.000  val loss 0.530 ± 0.029  val acc 0.790  nonzero weights 112
l1 λ=0.004 : train acc 1.000  val loss 0.514 ± 0.022  val acc 0.767  nonzero weights 51
l1 λ=0.008 : train acc 0.893  val loss 0.502 ± 0.012  val acc 0.768  nonzero weights 27
l1 λ=0.016 : train acc 0.772  val loss 0.532 ± 0.005  val acc 0.742  nonzero weights 10

(The vocabulary is rebuilt from each split’s training questions, so it averages 383 words rather than split 0’s 379.)

One caution before reading the table. Every run stops at 4,000 steps, so the “no penalty” row is not the limit it looks like. It’s early-stopped, and early stopping is itself a regularizer. Train that model 4× longer and its validation loss gets worse: How sure are we? shows L2 beating it by 0.131 at 16,000 steps, against 0.064 at 4,000. The two weakest L1 settings may not have converged either.

L2 traces the classic U. Validation loss falls from 0.480 with no penalty to 0.416 at $\lambda = 0.003$, then climbs as the penalty starts erasing real signal. At $\lambda = 0.1$ the model has decayed almost to “always predict the base rate”. Its training accuracy is 0.751, barely above the 0.75 that answer gets. Too little penalty overfits, and too much underfits, failing to learn even the training data.

Accuracy doesn’t follow the loss. This is the surprise in the table. The best-loss setting, $\lambda = 0.003$, has lower validation accuracy (0.794) than no penalty at all (0.818). Accuracy only asks which side of 0.5 each prediction falls on. Log-loss also asks how confident the prediction was. Unpenalized weights are huge, so the model’s wrong answers come with probabilities like 0.99. L2 mostly fixes that confidence without changing many decisions. Which of the two numbers to optimize depends on whether anything downstream uses the probability. For a router that falls back to a slower, more careful search when it’s unsure, the probability is the whole point. That’s the subject of Measuring a Classifier Honestly.

Notice also the ± column. It is the naive standard error of the 20-split mean, the spread across splits divided by $\sqrt{20}$ (How sure are we? shows it is too small, since the splits share data). Its size still tracks how much validation loss varies from split to split: ± 0.034 for the unpenalized model, falling steadily as $\lambda$ grows. That column is only a rough proxy for what regularization is for. It measures how much the result depends on which questions landed where. The variance in the bias–variance trade-off is narrower: how much the model’s prediction for a fixed question would move if it were trained on a different sample of 120 questions. Regularization trades a little bias (weights pulled away from what the data wants) for less of that variance, and the column shrinks for the same reason. Every regularizer, including early stopping, dropout, and data augmentation later in the series, makes that trade in some form.

L1 is sparser, and worse here. At $\lambda = 0.008$ it keeps 27 of 383 weights on average (30 on the split below) and reaches a validation loss of 0.502, clearly worse than L2’s 0.416. In fact every L1 setting is worse even than the early-stopped no-penalty row. With many weak, correlated words that each carry a little evidence, L1 keeps one word from each group and drops the rest, where L2 spreads the weight across all of them. (The elastic net adds both penalties to get sparsity without that loss.) L1’s choice is also unstable: which of two correlated words survives can flip from one split to the next. So the list below is one split’s answer, not a property of the data. Still, it’s the most revealing output in this post:

import numpy as np
from router_data import split, featurize, tokens
from overfit import sigmoid, log_loss, train

# 3. Which words does L1 keep? (one split, λ = 0.008)
((Xtr, ytr), _, _), vocab = featurize(*split(seed=0))
w, b = train(Xtr, ytr, 0.008, "l1")
kept = sorted((w[j], vocab[j]) for j in np.flatnonzero(w))
print(f"{len(kept)} of {len(vocab)} weights are nonzero; those of size >= 0.01:")
print("toward serving:", ", ".join(f"{t} {v:+.2f}" for v, t in kept[::-1] if v >= 0.01))
print("away:          ", ", ".join(f"{t} {v:+.2f}" for v, t in kept if v <= -0.01))

Full script: l1_words.py

30 of 379 weights are nonzero; those of size >= 0.01:
toward serving: batching +1.70, to +1.60, token +1.34, prefix +1.26, model +1.16, memory +1.14, decode +0.91, my +0.81, tokens +0.80, draft +0.74, one +0.68, vllm +0.63, request +0.61, for +0.50, does +0.33, on +0.28, throughput +0.22, cache +0.22, time +0.12
away:           agent -1.80, need -0.87, what -0.74, attention -0.39, how -0.37, should -0.12, layer -0.02

Half of these are what a person would pick: batching, prefix, decode, vllm, throughput toward serving, and agent, layer away. The other half are words like to, my, one, what and how, which carry no meaning about the topic. Some reflect mild phrasing habits: to turns up a little more often in serving questions, from questions like “How do I stream tokens to the browser?”. Others are pure accident. my is not a serving word across the whole dataset, but split 0’s training sample happened to put it mostly with serving, so this split learned my as serving evidence. With 120 training questions, the model can’t tell a real correlation from a lucky draw, and some of these would reverse on a different sample.

Two lessons in one output:

• Sparsity makes a model readable. A person can inspect 30 weights, not 379. That’s how we found the problem in the first place.
• Readable doesn’t mean right. What the weights say is what the data taught, including its accidents.

One good feature matches 379

The router from Learning Is Optimization used one feature: the count of 18 hand-picked serving words. How does it compare on exactly the same splits? compare.py imports the same helpers and runs all four feature sets on the 20 splits:

# 4. The hand-picked keyword count of "Learning Is Optimization" vs the learned bag of words.
SERVING = {"kv", "cache", "caching", "batch", "batching", "latency",
           "throughput", "serve", "serving", "vllm", "sglang", "tensorrt",
           "gpu", "decode", "prefill", "paged", "quantization", "memory"}
def hits(rs, words=SERVING):
    return np.array([[sum(t in words for t in tokens(r["question"]))]
                     for r in rs], dtype=float)
scores = {"keyword count": [], "keyword count, seen words only": [],
          "bag of words": [], "both": []}
for seed in range(20):
    tr, va, te = split(seed)
    ((Xtr, ytr), (Xva, yva), _), vocab = featurize(tr, va, te)
    seen = SERVING & set(vocab)             # keywords that occur in training
    designs = {"keyword count": (hits(tr), hits(va), 0.0),
               "keyword count, seen words only": (hits(tr, seen), hits(va, seen), 0.0),
               "bag of words": (Xtr, Xva, 0.003),
               "both": (np.c_[Xtr, hits(tr)], np.c_[Xva, hits(va)], 0.003)}
    for name, (A, B, lam) in designs.items():
        w, b = train(A, ytr, lam)
        p = sigmoid(B @ w + b)
        scores[name].append((log_loss(p, yva), ((p >= 0.5) == yva).mean()))
for name, s in scores.items():
    s = np.array(s)
    print(f"{name:31s} val loss {s[:, 0].mean():.3f}  val acc {s[:, 1].mean():.3f}")

Full script: compare.py

keyword count                   val loss 0.381  val acc 0.878
keyword count, seen words only  val loss 0.436  val acc 0.851
bag of words                    val loss 0.416  val acc 0.794
both                            val loss 0.347  val acc 0.866

On average the single keyword feature does at least as well as the best-regularized bag of words: 0.381 against 0.416 in loss, 0.878 against 0.794 in accuracy. It has just two parameters, a weight and a bias, so it has almost no room to overfit the training questions. (The list itself, though, was chosen by someone who had seen them.)

Where does its edge come from? The second row answers that. Restrict the keyword list to words that actually occur in each split’s training questions and it scores 0.436, worse than the bag of words. So the edge comes from keywords that appear in no training question at all, a few of the 18 in each split, like sglang or prefill. The bag of words can only learn weights for words it has seen, and a hand-written list can know about words the training data never showed. That’s the real lesson: with little data, knowledge built into a feature can supply what the data doesn’t contain.

Two cautions keep this honest.

• The edge is small and rests on a few questions. Those unseen keywords hit only a few of the 40 validation questions in each split. The next section shows that differences this size are within the noise.
• The same person wrote the keyword list and the questions, so this dataset may flatter the list.

Treat the comparison as an illustration of the principle, not proof of it. Real readers’ questions will be the proper test.

Both features together give the lowest loss, 0.347, though slightly lower accuracy (0.866) than the keywords alone. That accuracy difference is noise at this sample size. The keyword count carries prior knowledge, and the bag of words picks up what the list missed.

How sure are we?

Every comparison so far is an average over the same 20 splits. How big does a difference have to be before we believe it? The usual recipe is to compute the standard deviation of the 20 results and divide by $\sqrt{20}$. That assumes the 20 are independent, and they aren’t. Every split draws from the same 200 questions, so a question that happens to be hard for everyone pulls all 20 results the same way. Nadeau and Bengio worked out a correction for exactly this setting. Multiply the variance of the per-split differences by $\frac{1}{J} + \frac{n_\text{val}}{n_\text{train}}$ instead of $\frac{1}{J}$, where $J = 20$ splits. Here that makes the standard error 2.8 times larger. honest.py computes both:

# 6. How sure are we? Compare two settings on the same 20 splits (paired),
#    with the Nadeau-Bengio correction for splits that share questions.
def val_loss(lam, steps=4000):
    out = []
    for seed in range(20):
        ((Xtr, ytr), (Xva, yva), _), _ = featurize(*split(seed))
        w, b = train(Xtr, ytr, lam, steps=steps)
        out.append(log_loss(sigmoid(Xva @ w + b), yva))
    return np.array(out)

def compare(a, b, label):
    d = a - b
    naive = d.std(ddof=1) / np.sqrt(len(d))
    honest = d.std(ddof=1) * np.sqrt(1 / len(d) + 40 / 120)   # n_val / n_train
    print(f"{label:37s} {d.mean():+.3f}   naive SE {naive:.3f}   corrected SE {honest:.3f}")

l2 = val_loss(0.003)
compare(l2, val_loss(0.0), "L2 0.003 - no penalty (4,000 steps)")
compare(l2, val_loss(0.0, steps=16000), "L2 0.003 - no penalty (16,000 steps)")
compare(l2, val_loss(0.001), "L2 0.003 - L2 0.001")
compare(l2, val_loss(0.01), "L2 0.003 - L2 0.01")

Full script: honest.py

L2 0.003 - no penalty (4,000 steps)   -0.064   naive SE 0.020   corrected SE 0.055
L2 0.003 - no penalty (16,000 steps)  -0.131   naive SE 0.030   corrected SE 0.082
L2 0.003 - L2 0.001                   -0.005   naive SE 0.006   corrected SE 0.017
L2 0.003 - L2 0.01                    -0.019   naive SE 0.006   corrected SE 0.016

Read the corrected column and several conclusions soften.

• L2 versus no penalty. Against the early-stopped no-penalty model, L2’s gain is 0.064 ± 0.055, barely more than one standard error: suggestive, not decisive. Against the same model trained to 16,000 steps, the gain is 0.131 ± 0.082. That’s the fair comparison, because longer training is what “no penalty” means, and it’s the clearer result.
• The exact λ. 0.003 versus 0.001 or 0.01 is a tie. The U-shape is real, but its bottom is flat.
• The features. honest.py compares λ settings only. The feature gaps, 0.035 in mean loss between the keyword count and the bag of words and 0.034 between both together and the keywords alone, are just above the 0.03 rule of thumb below. Their paired error bars were not computed here, so this post does not call a winner between the feature sets.

The general rule for data this size: treat differences under about 0.03 in log-loss as ties, and report error bars that account for reused data. The shape of the results is solid: overfitting without a penalty, a U-shaped sweep, L1 sparsity. Most of the small margins between settings are not.

Don’t let the test set make decisions

What if we skip the validation set and pick $\lambda$ by the test set directly? To isolate the effect of where the choice is made, both sides use the same criterion, log-loss, and both report log-loss on the test set:

# 5. Choose λ by the test set instead of the validation set, using the SAME
#    criterion (log-loss) on both sides, and report test log-loss.
LAMBDAS = [0.0, 0.001, 0.003, 0.01, 0.03, 0.1]
peek = []
for seed in range(20):
    ((Xtr, ytr), (Xva, yva), (Xte, yte)), _ = featurize(*split(seed))
    fits = [train(Xtr, ytr, lam) for lam in LAMBDAS]
    test_loss = [log_loss(sigmoid(Xte @ w + b), yte) for w, b in fits]
    val_loss = [log_loss(sigmoid(Xva @ w + b), yva) for w, b in fits]
    peek.append((min(test_loss), test_loss[int(np.argmin(val_loss))]))
peek = np.array(peek)
print(f"test loss, λ chosen on test: {peek[:, 0].mean():.3f};"
      f" chosen on validation: {peek[:, 1].mean():.3f}")

Full script: peek.py

test loss, λ chosen on test: 0.403; chosen on validation: 0.439

Choosing on the test set reports a test loss of 0.403. Choosing honestly on validation and then measuring on test reports 0.439. The 0.036 gap isn’t a better model. It’s the selection fitting the noise in 40 test questions. Among six settings, one will do a little better on this particular test set by chance, and picking it bakes that luck into the reported number. Six settings on one split already inflate the result. A real project that tunes dozens of choices against the same test set can inflate it much further.

This is a form of leakage: information about the evaluation data reaching the model through a decision rather than through training. The common forms:

• tuning on the test set, as above;
• duplicates and near-duplicates across splits, such as the same question reworded, which make the test set partly a training set;
• preprocessing on all the data: computing the standardization mean, or a vocabulary with IDF weights, from rows that include the test set;
• features that exist only in hindsight, like a game’s final move count used to predict its winner.

Predicting Chess Outcomes encodes that last kind as a written contract that the pipeline enforces. The general rule is simple: every decision uses training and validation data only, and the test set is touched once.

Where this leads

• How good is the combined router, really? Accuracy treats every mistake the same, and a quarter of the questions are positive, so “always no” already scores 0.75. Measuring a Classifier Honestly measures the router with precision, recall and a confusion matrix, picks a threshold by what each mistake costs, and asks whether its probabilities can be trusted (calibration, which is exactly what L2 improved here). It also runs the router on the 20 Vietnamese questions, whose words aren’t in its vocabulary at all.
• Features the model builds itself. Both of this post’s lessons, that features cap a linear model and that a hand-built feature matched learned ones, point the same way. We want a model that learns its own features from raw input. That’s a neural network, and training one needs backpropagation (see Backpropagation, by Hand).
• Weight decay grows up. L2 is the weight decay that appears, in its corrected form, in the AdamW optimizer (see Initialization and Optimizers). Dropout (Dropout and Normalization) is another regularizer that makes the same bias–variance trade.
• Words as vectors. A bag of words treats token and tokens, or cache and caching, as unrelated. Embeddings and Contrastive Learning puts related words near each other, so knowledge about one transfers to the other.

Takeaways

• Features decide what a model can learn. A bag of words turns text into a 0/1 vector over the training vocabulary, and drops any word it hasn’t seen.
• Split the data by job. Training fits, validation decides, and test reports once. With small sets, average over many splits. The splits share data, though, so correct the standard error for it: here it was 2.8× too small.
• With more features than examples, the data is usually linearly separable. The log-loss then has no finite minimum, and gradient descent walks the weights toward infinity: training loss goes to 0 while validation loss turns back up. That is overfitting.
• L2 adds $\frac{\lambda}{2}\|w\|^2$, which is weight decay and a Gaussian prior. It restores a finite minimum and adds $\lambda$ to every curvature eigenvalue. Against unpenalized training run to 16,000 steps it cut validation loss by 0.131 ± 0.082. Early stopping at 4,000 steps got part of the way on its own; against that model, L2 cut the loss mostly by taming overconfidence rather than by changing decisions.
• L1 adds $\lambda\|w\|_1$, a Laplace prior. The soft-threshold step $\operatorname{sign}(v)\max(|v| - \eta\lambda, 0)$ sets most weights to exactly zero, which makes the model readable. Reading it exposed words like to and my that mean nothing. Here every L1 setting lost to L2, because L1 keeps one of each group of correlated weak words.
• One feature with domain knowledge (0.381) matched 383 learned ones (0.416, a 0.035 gap whose paired error bar was not computed). Its edge came from keywords that no training question contained. Combining both did best (0.347).
• Choosing a setting by the test set made the test loss look 0.036 better than an honest choice. That is leakage through a decision.

References

Start with Hastie, Tibshirani and Friedman §3.4 for ridge and lasso side by side, then Tibshirani’s lasso paper for why L1 selects features. Nadeau and Bengio and Cawley and Talbot are the two to read before trusting any cross-validation numbers.

• Hoerl and Kennard, Ridge Regression: Biased Estimation for Nonorthogonal Problems (Technometrics, 1970): L2-penalized regression.
• Tibshirani, Regression Shrinkage and Selection via the Lasso (JRSS B, 1996): the L1 penalty and why it selects features.
• Krogh and Hertz, A Simple Weight Decay Can Improve Generalization (NeurIPS, 1991): weight decay in neural networks.
• Parikh and Boyd, Proximal Algorithms (2014): the proximal step and soft-thresholding.
• Soudry et al., The Implicit Bias of Gradient Descent on Separable Data (JMLR, 2018): why unregularized logistic regression’s weights grow like $\log t$ on separable data.
• Cawley and Talbot, On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation (JMLR, 2010): why tuning on the evaluation data inflates the reported score.
• Nadeau and Bengio, Inference for the Generalization Error (Machine Learning, 2003): the corrected standard error for repeated train/test splits that share data.
• Zou and Hastie, Regularization and Variable Selection via the Elastic Net (JRSS B, 2005): why the lasso picks one of a group of correlated features, and the combined L1 + L2 penalty.
• Kaufman, Rosset, Perlich and Stitelman, Leakage in Data Mining (ACM TKDD, 2012): a taxonomy of leakage.
• Salton and Buckley, Term-Weighting Approaches in Automatic Text Retrieval (Information Processing & Management, 1988): bag-of-words weighting and TF-IDF.
• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2nd ed., 2009): shrinkage methods (§3.4) and cross-validation (§7.10).