Fine-Tuning, and What It Destroys

Build a Tiny GPT, End to End, and Train It on This Blog trained a model from nothing, and Small and Specialized Models used models someone else had trained. Most real training sits between the two: start from a pretrained model and keep training it on new data. That is fine-tuning. It is cheap compared with pretraining, because the model already knows language, and it is dangerous for the same reason: the weights that hold what it knows are the weights fine-tuning changes.

Table of contents

Three kinds of fine-tuning

All three use the same next-token cross-entropy as pretraining (Cross-Entropy and the Softmax Gradient). They differ in the data and in which tokens count:

• Continued pretraining: raw text from a new domain (legal filings, code in a new language, this blog), every token trained. The model learns the domain’s vocabulary, facts and style.
• Supervised fine-tuning (SFT), or instruction tuning: pairs of a prompt and a good answer, trained only on the answer. The model learns a behavior: to answer questions, follow a format, take a role. FLAN (Wei et al., 2022) showed that tuning on many tasks phrased as instructions makes a model better at instructions it never saw.
• Preference tuning: pairs of a better and a worse answer, which RLHF, DPO and GRPO and Reinforcement Learning from Verifiable Rewards cover, after Reinforcement Learning for Language Models sets up reinforcement learning.

ULMFiT (Howard and Ruder, 2018) set out the recipe that BERT and GPT made standard: pretrain on general text, fine-tune on the target domain, then on the target task. It protected the general knowledge with learning rates that differ by layer and rise then fall over training, and by unfreezing the layers one at a time from the top.

The SFT loss, with the prompt masked

A chat model never sees “a prompt” and “an answer” as separate things: a chat template turns the conversation into one token sequence with special tokens marking each turn. The SFT loss is ordinary next-token cross-entropy over that sequence, except that the positions whose target belongs to the prompt get the label $-100$, which PyTorch’s cross-entropy ignores. Worked on one example, with Qwen2.5’s template (loaded from the 1.5B Instruct model’s tokenizer; the 0.5B base model used below has the same vocabulary and template):

# Supervised fine-tuning (SFT) on one chat example, through Qwen2.5's chat
# template. The model reads the whole conversation, but only the assistant's
# tokens are trained: every other position gets the label -100, which
# PyTorch's cross_entropy skips (ignore_index=-100).
tok = AutoTokenizer.from_pretrained(snapshot_download("Qwen/Qwen2.5-1.5B-Instruct",
                                                      revision="989aa7980e4cf806f80c7fef2b1adb7bc71aa306"))
chat = [{"role": "user", "content": "What does the KV cache store?"},
        {"role": "assistant", "content": "The keys and values of every earlier token, so they aren't recomputed."}]
prompt = tok.apply_chat_template(chat[:1], tokenize=False, add_generation_prompt=True)
full = tok.apply_chat_template(chat, tokenize=False)
print(full)
p_ids, f_ids = tok(prompt).input_ids, tok(full).input_ids
assert f_ids[:len(p_ids)] == p_ids                        # the prompt is a prefix of the full text
labels = [-100] * len(p_ids) + f_ids[len(p_ids):]
print(f"{len(f_ids)} tokens: {len(p_ids)} of prompt (system, user, assistant header), masked; "
      f"{len(f_ids) - len(p_ids)} trained:")
print(repr(tok.decode(f_ids[len(p_ids):])))

# In the loss, input positions 0..n-2 predict targets 1..n-1, and targets
# of -100 drop out of the average: F.cross_entropy(logits, y, ignore_index=-100).
y = torch.tensor(labels[1:])
print(f"targets in the loss: {(y != -100).sum().item()} of {len(y)}")

Full script: sft_loss.py

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What does the KV cache store?<|im_end|>
<|im_start|>assistant
The keys and values of every earlier token, so they aren't recomputed.<|im_end|>

54 tokens: 36 of prompt (system, user, assistant header), masked; 18 trained:
"The keys and values of every earlier token, so they aren't recomputed.<|im_end|>\n"
targets in the loss: 18 of 53

Three details are worth noticing. The template adds a system prompt nobody wrote; train with it and serve with it, or the model sees at serving time a sequence it wasn’t trained on. Of 53 targets, 18 are trained: the model isn’t taught to predict the user’s question, which it should never generate. And the trained tokens include the end-of-turn token <|im_end|> and the newline after it: that is how the model learns to stop.

How far to move: learning rate, steps and memory

Pretraining a model of this size uses peak learning rates around $10^{-4}$ to $10^{-3}$. Fine-tuning usually uses 10 to 100 times less, for one to a few passes over the data, because the starting point is already good: the goal is a short move to a nearby point that also fits the new data. The experiment below measures what happens at $10^{-5}$ and $10^{-4}$.

Memory is the same as for pretraining. Updating every weight with AdamW in mixed precision costs 16 bytes per parameter before any activations (Initialization and Optimizers): 8 GB for the 0.5B model used here, 128 GB for an 8B model, more than one 80 GB GPU. That cost is what Parameter-Efficient Fine-Tuning’s LoRA removes. On the 16 GB Apple M4 laptop used here that left too little room for activations and the 151,936-wide logits, so the experiment freezes the tied embedding table (136 million of the 494 million weights), saving its gradients and AdamW state, 1.6 GB. Gradient checkpointing, used here, trades the activations’ memory for recomputing them in the backward pass.

An experiment: fine-tuning on this blog

Qwen2.5-0.5B, the base model, trained further on this blog’s posts: the training split of the blog corpus as the data script of Sequences Before Attention rebuilt it on 2026-10-04 (1,747,277 characters), about 530,000 tokens, with two posts held out. Each step takes 4 windows of 512 tokens at random, for 400 steps, about 1.5 passes over the data, with AdamW and a 20-step warmup. Three measurements, in bits per byte, at each checkpoint:

• the held-out blog posts, what fine-tuning should improve;
• WikiText-2’s test set, its first 120,000 characters (7 articles), general English the model knew before;
• Vietnamese Wikipedia, something else it knew, which no run trains on.

Two learning rates, $10^{-5}$ and $10^{-4}$, each with and without replay: one window in four drawn from English Wikipedia instead of the blog. Two seeds each.

The replay text needs care. A first version of this experiment replayed WikiText’s own training split, and on WikiText’s test set the replaying model ended better than it started, 0.981 down to 0.905 bits per byte: it was learning WikiText’s peculiar formatting (” @-@ ” for hyphens, ”= = Heading = =” for sections), which the test shares, not remembering English. Here replay draws from raw Wikipedia, excluding any article with a test-set title (none were in this shard). The two Wikipedia files are single shards of the wikimedia/wikipedia dataset’s November 2023 dump, 20231101.en/train-00001-of-00041.parquet and 20231101.vi/train-00002-of-00004.parquet, which fetch_wiki.py downloads as wiki/en.parquet and wiki/vi.parquet.

Full script: fetch_wiki.py

wiki_data.py drops the test’s articles from the English shard and cuts the replay text and the Vietnamese test from the two files. WIKI is the folder of WikiText-2 (raw), downloaded at a pinned revision, and the first line reads its test set:

test = "".join(pd.read_parquet(WIKI + "test-00000-of-00001.parquet")["text"])[:120_000]
titles = {re.sub(r" @(.)@ ", r"\1", t.strip()).replace("( ", "(").replace(" )", ")")   # undo WikiText's spacing
          for t in re.findall(r"^ = ([^=].*?) = $", test, flags=re.M)}
en = pd.read_parquet("wiki/en.parquet", columns=["title", "text"])
dropped = int(en["title"].isin(titles).sum())
en = en[~en["title"].isin(titles)].sample(frac=1.0, random_state=0)
replay, n = [], 0
for t in en["text"]:
    replay.append(t); n += len(t)
    if n > 4_000_000:
        break
open("wiki_replay.txt", "w").write("\n\n".join(replay))
vi = pd.read_parquet("wiki/vi.parquet", columns=["text"])
vi = vi[vi["text"].str.len() > 3000].sample(frac=1.0, random_state=0)      # skip the many one-line stubs
out, n = [], 0
for t in vi["text"]:
    out.append(t); n += len(t)
    if n > 120_000:
        break
open("vi_test.txt", "w").write("\n\n".join(out)[:120_000])
print(f"WikiText test articles: {len(titles)}, of which {dropped} are in this shard and dropped; replay text {sum(map(len, replay)):,} chars from {len(replay)} articles")

Full script: wiki_data.py

WikiText test articles: 7, of which 0 are in this shard and dropped; replay text 4,001,877 chars from 1101 articles

ft.py loads the model with its embedding table frozen and gradient checkpointing on, and scores the three texts in bits per byte at each checkpoint. Its batches and training loop:

def batch():
    n_wiki = round(B * replay)
    rows = []
    for k in range(B):
        src = wiki_ids if k < n_wiki else blog_ids
        i = torch.randint(0, len(src) - T - 1, (1,), generator=g).item()
        rows.append(src[i:i + T + 1])
    return torch.stack(rows).to(DEVICE)

log = {}
t0 = time.perf_counter()
for step in range(steps + 1):
    if step in checkpoints:
        log[step] = {k: round(bits_per_byte(*v), 4) for k, v in evals.items()}
    if step == steps:
        break
    x = batch()
    for group in opt.param_groups:                                  # 20-step warmup, then constant
        group["lr"] = lr * min(1.0, (step + 1) / 20)
    opt.zero_grad(set_to_none=True)
    for half in (x[:B // 2], x[B // 2:]):                            # two micro-batches of 2: the same batch of 4,
        with torch.autocast(DEVICE, dtype=torch.bfloat16):          # in half the memory for activations and logits
            logits = model(half[:, :-1]).logits
        loss = F.cross_entropy(logits.float().reshape(-1, logits.shape[-1]), half[:, 1:].reshape(-1)) / 2
        loss.backward()
        del logits, loss
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()

Full script: ft.py

ft.py runs as python ft.py <lr> <replay> 400 <seed> and prints one summary line per run. finetune_all.py runs all eight (four settings, two seeds) and keeps those lines in ft.out, and ft_table.py turns them into this table:

Full script: finetune_all.py

Full script: ft_table.py

before training: blog 1.163, WikiText 0.981, Vietnamese 1.022 bits per byte

bits per byte after each step, seed 0 then seed 1
lr      replay         step 25       step 50       step 100      step 200      step 400
1e-05     0%  blog     1.112 1.113   1.086 1.092   1.072 1.072   1.069 1.065   1.068 1.084
1e-05     0%  wiki     0.990 0.990   1.002 1.004   1.012 1.011   1.031 1.026   1.047 1.050
1e-05     0%  vi       1.028 1.029   1.035 1.035   1.042 1.042   1.054 1.052   1.071 1.072
1e-05    25%  blog     1.118 1.117   1.094 1.097   1.077 1.079   1.072 1.066   1.060 1.068
1e-05    25%  wiki     0.985 0.982   0.988 0.988   0.989 0.989   0.996 0.993   0.997 0.997
1e-05    25%  vi       1.022 1.022   1.021 1.020   1.021 1.021   1.024 1.022   1.026 1.025
0.0001    0%  blog     1.197 1.214   1.219 1.217   1.247 1.241   1.256 1.259   1.311 1.357
0.0001    0%  wiki     1.124 1.101   1.159 1.151   1.253 1.232   1.371 1.338   1.460 1.513
0.0001    0%  vi       1.114 1.105   1.149 1.136   1.251 1.223   1.400 1.506   1.502 1.609
0.0001   25%  blog     1.193 1.210   1.228 1.230   1.254 1.228   1.263 1.238   1.294 1.292
0.0001   25%  wiki     1.082 1.111   1.165 1.165   1.239 1.219   1.291 1.284   1.320 1.351
0.0001   25%  vi       1.088 1.088   1.146 1.178   1.239 1.226   1.572 1.293   1.455 1.400

At $10^{-5}$ the model learns the blog, and forgets as it goes. The held-out posts improve from 1.163 to about 1.07 bits per byte, most of it in the first 100 steps. WikiText gets steadily worse, by 0.03 at step 100 and 0.07 at step 400, and so does Vietnamese, by 0.02 and 0.05, though nothing in training is Vietnamese or touches it on purpose. After step 200 the gain stops (one seed even loses some, 1.065 to 1.084, once it has seen about one pass’s worth of tokens and windows increasingly repeat) while the forgetting keeps growing. Stopping at step 100 would have kept nearly all of the gain and under half of the forgetting. (Step 100 is picked here by looking at the same held-out posts; in practice it would be chosen on a separate validation split.)

At $10^{-4}$ it learns nothing and breaks. The blog gets worse from the first checkpoint, 1.163 to 1.20 by step 25 and 1.31 to 1.36 by step 400, and WikiText and Vietnamese lose about half a bit per byte. $10^{-4}$ is at the low end of the peak learning rates used to pretrain models of this size, and even that throws the model out of the region where its knowledge lives. Two reasons it’s too large here, besides the starting point already being good: pretraining schedules end with the learning rate decayed far below its peak, so the model was last trained with much smaller steps, and these batches are 2,048 tokens, against millions in pretraining, so each step’s gradient is much noisier, while Adam still moves each weight by up to about the learning rate per step. The short warmup, 20 steps, doesn’t help.

Replay at $10^{-5}$ removes most of the forgetting, and by step 400 costs nothing on the blog. With one window in four from Wikipedia, WikiText loses 0.016 at step 400 instead of 0.07, and Vietnamese loses 0.004 instead of 0.05. The blog learns a little more slowly at first, seeing a quarter fewer blog windows, but ends at 1.060 and 1.068, as good as without replay. That isn’t just the smaller share of blog text: a run without replay had already lost 0.03 on Vietnamese by step 200, after fewer blog windows than the 300 steps’ worth the replay run sees in 400.

Vietnamese was protected too, though none was replayed. So replay isn’t only rehearsing the facts it replays: the general text in each batch pulls back against the changes that would hurt general text, and those changes hurt Vietnamese as well. Both tests are Wikipedia, like the replay, so part of this may be rehearsing the genre (encyclopedic prose, names, dates); a test outside Wikipedia would tell the two apart.

Replay can’t rescue a learning rate that is too high. At $10^{-4}$ it helps a little (WikiText 1.32 and 1.35 instead of 1.46 and 1.51 at step 400) but every measurement still ends far worse than it started, and erratically: one seed’s Vietnamese score reaches 1.57 at step 200 before coming back to 1.46.

Two caveats. The tied embedding table, 136 million of the 494 million weights, was frozen to fit training on this laptop, so these are full fine-tunes of everything else. And this is one small model, one small dataset, a held-out blog of two posts and two seeds per setting; the seeds agree to within about 0.02 at $10^{-5}$ and disagree more at $10^{-4}$, where training is unstable.

Why it forgets

McCloskey and Cohen (1989) named it catastrophic interference: a network trained on task A and then only on task B loses A, often abruptly, because the same weights store both and B’s gradient has no reason to protect what A needed. Fine-tuning is exactly that setup. Nothing in the loss mentions WikiText; every step moves weights toward the blog, and some of those weights were doing other work.

Three families of remedies:

• Replay: keep training on some of the old distribution. It is the simplest and, as measured above, it works, at the price of having data like the original training set and spending part of each batch on it.
• Regularization: penalize moving the weights that mattered for the old task. EWC below is the classic form; the simplest form is a small learning rate and few steps.
• Parameter isolation: freeze the original weights and train new ones next to them. LoRA and adapters (Parameter-Efficient Fine-Tuning) work this way, and the original model is always one step away: drop the new weights.

EWC, derived

Elastic weight consolidation (Kirkpatrick et al., 2017) asks which weights mattered for task A, and pulls those back. Start from Bayes’ rule for weights $\theta$ after seeing data $A$ and then $B$:

The first term is the new task’s log-likelihood, the negative of its loss. The second holds everything task $A$ taught. It can’t be computed exactly, so approximate it by a second-order Taylor expansion around $\theta^*_A$, the weights after training on $A$ (the Laplace approximation: the posterior as a Gaussian around its peak; with a flat prior, that peak is the minimum of $A$‘s loss). At that minimum the first derivative is zero, so

with $H$ the curvature of $-\log p(\theta \mid A)$ at $\theta^*_A$. For a model trained by maximum likelihood, $H$ is approximated by the Fisher information $F$, the expected outer product of the gradient of the log-probability, which only needs gradients ($H \approx N F$ for $N$ examples; the $N$ folds into $\lambda$ below). Keeping only its diagonal, one number per weight, gives the EWC loss for training on $B$:

$F_i$ is the average, over task-$A$ examples, of the squared gradient of $\log q(y \mid x)$ with respect to weight $i$, for one example at a time (not the square of a batch’s average gradient), with $y$ sampled from the model. Using the data’s own labels for $y$ instead gives the “empirical Fisher”, the common cheap substitute, and a worse approximation of $H$. A weight whose change barely moved task $A$‘s likelihood has $F_i \approx 0$ and is free to move; a weight task $A$ depended on is held near its old value by a spring of stiffness $\lambda F_i$. $\lambda$ sets how much of the old task to keep. For a pretrained language model, “task $A$” is pretraining, so $F$ has to be estimated from per-example gradients on a sample of general text, and stored along with a copy of the original weights: two more numbers per parameter.

When fine-tuning is the wrong tool

Fine-tuning changes how a model behaves: its format, its style, its domain vocabulary, the kind of answer it gives. It is a poor way to give a model facts it must state reliably. A fact seen a few times in fine-tuning data is learned weakly, it can’t be updated without training again, and the model can’t say where it came from. Ovadia et al. (2024) compared the two on knowledge-heavy questions and found retrieval beat unsupervised fine-tuning, both for facts the model had seen and for new ones. For facts, put them in the context: retrieval, from Vector Search on. For behavior, fine-tune.

Where this leads

• Parameter-Efficient Fine-Tuning fine-tunes with a tiny fraction of the parameters, LoRA, and measures what that does to forgetting.
• Knowledge Distillation distills a large model into a small one, and Reinforcement Learning for Language Models through GRPO and Reinforcement Learning from Verifiable Rewards tune for preferences and rewards instead of next tokens.

Takeaways

• Fine-tuning is next-token training continued on new data; SFT trains only the answer’s tokens, the prompt’s labels masked to $-100$, and the end-of-turn token is what teaches the model to stop.
• The learning rate decides what fine-tuning does. At $10^{-5}$ this model learned the blog in about 100 steps; at $10^{-4}$, near pretraining’s peak, it got worse at everything from the start.
• Fine-tuning forgets what it doesn’t train on, including things unrelated to the new data (Vietnamese, here), and keeps forgetting after it stops learning, so stop early, choosing when on a validation split.
• Replay, a quarter of each batch from general text, cut the forgetting to a quarter on English and to almost nothing on Vietnamese, which it never replayed, at no cost to the final gain: it holds back more than just what it rehearses.
• Check that replay data doesn’t share the test set’s distribution: a first version replayed WikiText’s training split and made WikiText look better than before fine-tuning.
• EWC is the principled alternative: a quadratic penalty on moving each weight, weighted by the diagonal Fisher information from the old task.
• Fine-tune for behavior and style; for facts the model must state and update, retrieval works better.

References

Start with Kirkpatrick et al. for EWC, then Ovadia et al. on fine-tuning against retrieval for facts. Howard and Ruder’s ULMFiT paper is the origin of the pretrain-then-fine-tune recipe.

• Howard and Ruder, Universal Language Model Fine-tuning for Text Classification (ACL, 2018): the pretrain, then fine-tune recipe.
• Wei et al., Finetuned Language Models Are Zero-Shot Learners (ICLR, 2022): FLAN, instruction tuning.
• McCloskey and Cohen, Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem (Psychology of Learning and Motivation, 1989): forgetting, named as catastrophic interference.
• Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks (PNAS, 2017): elastic weight consolidation, derived here.
• Ovadia, Brief, Mishaeli and Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (EMNLP, 2024): retrieval against fine-tuning for facts.
• Qwen Team, Qwen2.5 Technical Report (2024): the base model fine-tuned here.