Parameter-Efficient Fine-Tuning: LoRA, Prefix and Prompt Tuning

Fine-Tuning, and What It Destroys fine-tuned nearly every weight of a 0.5B model: 16 bytes of training state per parameter, and a full copy of the model for every fine-tune kept. Most of that is often unnecessary. The change a task needs can usually be made with far fewer degrees of freedom than a weight matrix has entries, and parameter-efficient fine-tuning trains only those: a fraction of a percent of the parameters, with the original weights frozen.

Table of contents

Is the update low-rank?

Aghajanyan et al. (2021) asked how many parameters fine-tuning really needs. They froze a pretrained model and trained only a small vector of $d$ numbers, mapped into the full weight space by a fixed random projection, and raised $d$ until the fine-tuned model reached 90% of full fine-tuning’s score. For RoBERTa on the MRPC paraphrase task, about 200 numbers were enough. They called the smallest such $d$ the task’s intrinsic dimension, and found it shrinks as pretraining goes on and as models get larger: a better pretrained model needs a smaller move.

LoRA (Hu et al., 2022) is inspired by it, with a different restriction: not a random subspace of all the weights, but a low rank for the update to each weight matrix, learned rather than fixed. A matrix of rank $r$ is a sum of $r$ outer products, one per direction it acts in. If the update a task needs, $\Delta W$, has most of its size in a few directions, a rank-$r$ matrix with small $r$ can carry it. The widget shows a $24 \times 24$ update and the best approximation of it at each rank, the most any rank-$r$ update could capture (the Eckart–Young theorem: truncate the singular value decomposition). Compare an update with a few strong directions to a random one:

With three strong directions, rank 3 captures 99% of the update’s squared size with 144 numbers instead of 576. A random update has no such structure: rank 3 captures under 40% of it. LoRA bets that the update a task needs can look like the first case. It isn’t always so: Biderman et al. (2024) found full fine-tunes on large amounts of code and math making updates of 10 to 100 times the ranks LoRA usually uses, with LoRA falling behind. The experiment below checks the bet on a small fine-tune, by measuring the rank of a full fine-tune’s update and by training LoRA at several ranks.

LoRA

For a frozen pretrained weight $W$ of shape $d_\text{out} \times d_\text{in}$, LoRA adds a trainable update in factored form:

• Shapes. $A$ maps the input down to $r$ numbers, $B$ maps them back up. The update $BA$ has the shape of $W$ and rank at most $r$, but is stored as $r(d_\text{in} + d_\text{out})$ numbers instead of $d_\text{in} d_\text{out}$. For a $4096 \times 4096$ matrix at $r = 16$: 131,072 numbers instead of 16.8 million, 0.8%.
• Initialization. $A$ is random and $B$ is zero, so $BA = 0$ and training starts from exactly the pretrained model. Both can’t be zero: the gradient of $B$ is proportional to $A x$ and the gradient of $A$ to $B$, so two zero matrices would never move.
• Scale. $\alpha / r$ multiplies the update. With Adam, each step moves each entry by at most about the learning rate, whatever the gradient’s size (gradient clipping doesn’t change that: Adam divides the clipping factor back out), so the scale is what sets how fast $BA$ grows. Fixing $\alpha$ while changing $r$, as the LoRA paper does, removes the need to retune the learning rate for every rank. A common alternative, $\alpha = 2r$, keeps $\alpha / r$ at 2 instead, and the experiment below uses it and shows what it costs.
• Where. The paper applied it to the attention projections, mostly $W_q$ and $W_v$; applying it to every linear layer, the MLP’s included, is now common and is what the experiment does.
• Merging. After training, compute $W' = W + \frac{\alpha}{r} BA$ once. The merged model is an ordinary model with the original shapes, so serving it costs exactly what serving the base model costs. Or keep $B$ and $A$ separate and swap them per request, below.

Memory, worked for an 8B model

# LoRA's parameter and memory arithmetic for Llama 3.1 8B (its config.json:
# hidden 4096, MLP 14336, 32 layers, 32 query heads and 8 key/value heads of
# 128, vocabulary 128256). LoRA at rank r on a d_out x d_in matrix trains
# r * (d_in + d_out) numbers instead of d_in * d_out.
D, F, L, KV, V = 4096, 14336, 32, 8 * 128, 128256
mats = {"q": (D, D), "k": (D, KV), "v": (D, KV), "o": (D, D), "gate": (D, F), "up": (D, F), "down": (F, D)}
per_layer = sum(i * o for i, o in mats.values())
total = L * per_layer + 2 * V * D + (2 * L + 1) * D       # + input and output embeddings (untied), norm gains
print(f"Llama 3.1 8B: {total / 1e9:.2f}B parameters, {L * per_layer / 1e9:.2f}B of them in the 7 matrices of each block")
for r in (1, 8, 16, 64):
    lora = L * sum(r * (i + o) for i, o in mats.values())
    print(f"  LoRA r = {r:2d} on all 7: {lora / 1e6:6.1f}M trainable, {lora / total:.2%} of the model")
r16 = L * sum(16 * (i + o) for i, o in mats.values())
GB = 1e9
print("\nMemory for training, weights + gradients + AdamW's two moments (activations not counted):")
print(f"  full fine-tune, mixed precision (16 bytes per parameter, Initialization and Optimizers): {16 * total / GB:6.1f} GB")
print(f"  LoRA r = 16: 16-bit frozen weights {2 * total / GB:.1f} GB + 16 bytes per LoRA parameter "
      f"{16 * r16 / GB:.2f} GB = {(2 * total + 16 * r16) / GB:6.1f} GB")
q4 = 0.5 * L * per_layer + 2 * (total - L * per_layer)          # 4-bit block matrices, 16-bit embeddings and norms
print(f"  QLoRA r = 16: 4-bit block weights, the rest 16-bit {q4 / GB:.1f} GB + LoRA {16 * r16 / GB:.2f} GB "
      f"= {(q4 + 16 * r16) / GB:6.1f} GB")
dq = 0.127 / 8 * L * per_layer                                   # QLoRA's double-quantized block scales
print(f"  (plus NF4's block scales: 0.127 bits per weight with double quantization, {dq / GB:.2f} GB)")
print(f"\nOne r = 16 adapter stored in 16 bits: {2 * r16 / 1e6:.0f} MB, against {2 * total / GB:.1f} GB for a full copy")

Full script: memory.py

Llama 3.1 8B: 8.03B parameters, 6.98B of them in the 7 matrices of each block
  LoRA r =  1 on all 7:    2.6M trainable, 0.03% of the model
  LoRA r =  8 on all 7:   21.0M trainable, 0.26% of the model
  LoRA r = 16 on all 7:   41.9M trainable, 0.52% of the model
  LoRA r = 64 on all 7:  167.8M trainable, 2.09% of the model

Memory for training, weights + gradients + AdamW's two moments (activations not counted):
  full fine-tune, mixed precision (16 bytes per parameter, Initialization and Optimizers):  128.5 GB
  LoRA r = 16: 16-bit frozen weights 16.1 GB + 16 bytes per LoRA parameter 0.67 GB =   16.7 GB
  QLoRA r = 16: 4-bit block weights, the rest 16-bit 5.6 GB + LoRA 0.67 GB =    6.3 GB
  (plus NF4's block scales: 0.127 bits per weight with double quantization, 0.11 GB)

One r = 16 adapter stored in 16 bits: 84 MB, against 16.1 GB for a full copy

The savings are in the training state, not the weights: the frozen model still has to be in memory, but its gradients, the 32-bit master copy of its weights and AdamW’s two moments, 14 of the 16 bytes per parameter (Initialization and Optimizers), now exist only for the LoRA matrices. An 8B fine-tune goes from 128 GB, two 80 GB GPUs before activations, to 17 GB, one consumer-sized GPU. And a kept fine-tune is an 84 MB file, not a 16 GB copy of the model.

QLoRA (Dettmers et al., 2023) goes further: store the frozen weights in 4 bits, in a format called NF4 designed for normally distributed weights, dequantize each matrix to 16 bits when it’s used, and train 16-bit LoRA matrices on top. Gradients flow through the quantized weights to the LoRA matrices but never update the 4-bit numbers. That brings the 8B model to about 6 GB, and let the paper fine-tune a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning’s results. Quantization from Scratch covers quantization itself. The paper also quantizes the quantization constants themselves (“double quantization”), and pages optimizer state to CPU memory when the GPU runs short.

An experiment: LoRA on the earlier full fine-tune

Fine-Tuning, and What It Destroys’s setup, changed only where LoRA changes it: Qwen2.5-0.5B (base), 400 steps of 4 windows of 512 tokens drawn at random from this blog’s posts (about 1.5 passes over the 530,000 training tokens), AdamW with a 20-step warmup, and two measurements at each checkpoint, in bits per byte (Sequences Before Attention): held-out blog posts, which fine-tuning should improve, and WikiText-2, which stands in for everything the model knew before and should not get worse. The blog text, blog_train.txt and blog_val.txt, is the same as in that post: written by the data script of Sequences Before Attention, run on the posts as they stood on 2026-10-04. Every pretrained weight is frozen; each of the seven linear layers in each of the 24 blocks gets its own $A$ and $B$. Three ranks at a learning rate of $10^{-3}$, rank 8 again at $2 \times 10^{-4}$, two seeds each; then, after the first results, rank 64 at $1.25 \times 10^{-4}$ (two seeds) and rank 1 at $2 \times 10^{-4}$ (one seed). The last two rows are Fine-Tuning, and What It Destroys’s full fine-tune at $10^{-5}$, on the same batches, for comparison.

lora.py is one run, with the rank, learning rate and seed as arguments. The adapter is a wrapper around each frozen linear layer:

class LoRALinear(torch.nn.Module):
    def __init__(self, base, r, alpha):
        super().__init__()
        self.base, self.scale = base, alpha / r
        self.A = torch.nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
        self.B = torch.nn.Parameter(torch.zeros(base.out_features, r))
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T
    def merged(self):
        """W + (alpha / r) B A: one ordinary matrix again, no extra cost at inference."""
        return self.base.weight + self.scale * self.B @ self.A

and every pretrained weight is frozen before the seven projections of each block are wrapped:

torch.manual_seed(seed)
model = AutoModelForCausalLM.from_pretrained(PATH, dtype=torch.float32).to(DEVICE)
for p in model.parameters():
    p.requires_grad_(False)
names = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
for layer in model.model.layers:
    for parent in (layer.self_attn, layer.mlp):
        for n in names:
            if hasattr(parent, n):
                setattr(parent, n, LoRALinear(getattr(parent, n), rank, alpha).to(DEVICE))
model.gradient_checkpointing_enable()
model.enable_input_require_grads()                # checkpointing needs a grad path into the frozen model

Full script: lora.py

The rest of the script is the training loop and the measurements of Fine-Tuning, and What It Destroys, and at the end it merges $BA$ into $W$ and checks that the logits match. lora_runs.py runs it eleven times, for the settings above, keeping one summary line per run. full_ft.py reruns the full fine-tune with that post’s ft.py, after fetch_wiki.py and wiki_data.py have prepared its Wikipedia text, and lora_table.py turns the summary lines into the table:

Full script: lora_runs.py

Full script: full_ft.py

Full script: lora_table.py

before training: blog 1.163, WikiText 0.981 bits per byte

bits per byte after each step, seed 0 then seed 1 (rank 1 at 2e-4: one seed)
rank  lr            step 25       step 50       step 100      step 200      step 400
   1  0.001     blog  1.118 1.117   1.097 1.101   1.088 1.089   1.090 1.087   1.088 1.100
   1  0.001     wiki  1.000 1.000   1.018 1.022   1.029 1.034   1.056 1.051   1.074 1.084
   8  0.001     blog  1.113 1.111   1.100 1.100   1.099 1.101   1.118 1.116   1.150 1.179
   8  0.001     wiki  1.032 1.033   1.053 1.062   1.087 1.075   1.142 1.149   1.211 1.229
  64  0.001     blog  1.244 1.260   1.405 1.306   1.446 1.415   1.474 1.425   1.492 1.495
  64  0.001     wiki  1.214 1.198   1.437 1.298   1.594 1.494   1.767 1.687   1.965 2.001
   8  0.0002    blog  1.119 1.117   1.096 1.100   1.083 1.083   1.082 1.074   1.075 1.084
   8  0.0002    wiki  0.998 0.997   1.012 1.016   1.025 1.027   1.050 1.045   1.066 1.074
  64  0.000125  blog  1.117 1.118   1.092 1.094   1.083 1.083   1.087 1.082   1.095 1.123
  64  0.000125  wiki  1.043 1.043   1.055 1.049   1.063 1.061   1.102 1.092   1.135 1.147
   1  0.0002    blog  1.140         1.118         1.102         1.090         1.073
   1  0.0002    wiki  0.981         0.989         1.000         1.020         1.036
full, 1e-5      blog  1.112 1.113   1.086 1.092   1.072 1.072   1.069 1.065   1.068 1.084
full, 1e-5      wiki  0.990 0.990   1.002 1.004   1.012 1.011   1.031 1.026   1.047 1.050

seconds per 400-step run: 1481 to 2100
merged vs unmerged, largest logit difference over all 11 LoRA runs: 6.2e-04

Two seeds per setting, and a seed changes both $A$‘s starting values and the order of the training windows; at the same setting the seeds land 0.003 to 0.03 bits per byte apart on the blog by step 400. Rank and learning rate are also tangled here, as the next paragraphs show, so compare runs with that in mind.

Rank 1 learns about as much as any LoRA run. 0.55 million trainable parameters, 0.11% of the model, take the blog from 1.163 to 1.088 bits per byte by step 100, and at a lower learning rate to 1.073 by step 400 (one seed), within 0.01 of the full fine-tune’s best, about the gap between two seeds. For this fine-tune, one direction per matrix carries nearly all the gain LoRA gets at any rank. Full fine-tuning does a little better, 1.065 to 1.068 at its best.

At a fixed learning rate, the higher ranks overshoot, because of how the scale was set. At $10^{-3}$, rank 1 trains; rank 8 improves the blog until about step 100 and then gets worse, while WikiText gets worse throughout, faster than at rank 1; and rank 64 is worse than the untrained model from the first checkpoint: by step 400 the blog is at 1.49 bits per byte and WikiText at 2.0, double where it started. Here $\alpha = 2r$, so $\alpha / r = 2$ at every rank, and $BAx$ is a sum of $r$ terms whose entries Adam each moves by up to the learning rate per step. How much the sum grows with $r$ depends on how aligned the terms are: like $\sqrt{r}$ if their directions are unrelated, like $r$ if they all point the same way. So rank 64’s update moves 8 to 64 times as fast as rank 1’s. The LoRA paper fixes $\alpha$ instead, a scale of $1/r$, the fully aligned case, so it needn’t retune per rank; Kalajdzievski (2023) argues that over-corrects, slowing the higher ranks until they learn no more than low ones, and scales by $\alpha / \sqrt{r}$ (rsLoRA). Rank 8 at a fifth of the learning rate is the best two-seed run here, 1.075 and 1.084 at step 400.

With a smaller step, rank 64 trains. Scaling rank 1’s learning rate by $1/\sqrt{r}$ gives $10^{-3}/8 = 1.25 \times 10^{-4}$ for rank 64. At that rate both seeds reach 1.083 bits per byte by step 100, as good as rank 8 at the same step, so the first failure was the step size, not the rank. Then they overfit: by step 400 the blog is back up to 1.095 and 1.123 and WikiText is at 1.135 and 1.147, more forgetting than any run except those that overshot at $10^{-3}$. Sixty-four times the parameters bought nothing on this data, and cost more forgetting. The gentlest run is the other way round: rank 1 at $2 \times 10^{-4}$ (one seed) learns more slowly but is still improving at step 400, at 1.073, with WikiText at 1.036, the least forgetting of any run that learned.

LoRA forgets too. Two settings run with both seeds, rank 1 at $10^{-3}$ and rank 8 at $2 \times 10^{-4}$, gained 0.06 to 0.09 bits per byte on the blog by step 400 and lost 0.08 to 0.10 on WikiText, and the forgetting keeps growing after the blog stops improving: rank 1’s blog score is flat from step 100 to 400 while its WikiText score climbs from 1.03 to 1.08. In both runs, stopping at step 100 would have kept nearly all of the gain and about half of the forgetting. (Step 100 is picked here by looking at the same held-out posts the gain is measured on; in practice it would be chosen on a separate validation set.)

Against full fine-tuning, LoRA forgot more here, not less. Fine-Tuning, and What It Destroys’s full fine-tune (with its embedding table frozen to fit the laptop, so “full” means 358 million of 494 million weights) is best compared with LoRA at the same gain, since every run here keeps forgetting after it stops learning. At step 100 it has gained 0.091 bits per byte on the blog in both seeds, for 0.030 lost on WikiText. LoRA’s runs that came closest lost more for less: rank 8 at $2 \times 10^{-4}$ gained 0.080 by step 100 and lost 0.045, rank 1 at $10^{-3}$ gained 0.075 and lost 0.05, and rank 1 at $2 \times 10^{-4}$ needed until step 400 to gain 0.090, and lost 0.055. That is the opposite of Biderman et al. (2024), who found that LoRA learns less and forgets less than full fine-tuning, at larger scale. This is one small dataset, one learning rate for full fine-tuning and two seeds, and learning rates aren’t comparable across methods, since a LoRA weight and an original weight move by different amounts per step. What does carry over is that the step size decides the forgetting: Fine-Tuning, and What It Destroys’s full fine-tune at $10^{-4}$ failed the way rank 64 did at $10^{-3}$, worse on both measurements from the start.

But the full fine-tune’s own update is not low-rank. Rank 1 was enough for LoRA, so is the change full fine-tuning makes also close to rank 1? Fine-Tuning, and What It Destroys’s run was repeated with the fine-tuned weights saved (full_ft.py gives its ft.py a file name, ft_weights.pt, to write the block weights to), and each matrix’s update $\Delta W = W_\text{fine-tuned} - W_\text{pretrained}$ decomposed, as in the widget. The loop of spectrum.py:

for n in NAMES:
    keys = [k for k in tuned if k.endswith(f"{n}.weight")]
    rows = []
    for k in keys:
        s = torch.linalg.svdvals(tuned[k].double() - base[k].double())
        need = int((torch.cumsum(s ** 2, 0) / (s ** 2).sum() < 0.9).sum()) + 1
        rows.append([share(s, k_) for k_ in KS] + [need])
    shape = tuple(tuned[keys[0]].shape)
    r = torch.linalg.svdvals(torch.randn(shape, dtype=torch.float64))
    avg = torch.tensor(rows).mean(0)
    print(f"{n:10s} {shape[0]:>4d}x{shape[1]:<5d}   " + "   ".join(f"{x:8.1%}" for x in avg[:3]) +
          f"   {avg[3]:6.0f} of {min(shape)}   {share(r, 8):6.1%} / {share(r, 64):5.1%}")

Full script: spectrum.py

update from the full fine-tune of Fine-Tuning, and What It Destroys (lr 1e-5, no replay, 400 steps, seed 0), averaged over the 24 blocks;
share of the update's squared size captured by the best rank-k matrix, and the rank needed for 90%

matrix          shape   rank 1     rank 8     rank 64    rank for 90%   random matrix, rank 8 / 64
q_proj      896x896         6.6%      25.6%      63.6%      228 of 896     3.4% / 23.6%
k_proj      128x896        13.6%      45.9%      91.2%       60 of 128    11.0% / 65.9%
v_proj      128x896         6.9%      26.3%      80.3%       86 of 128    10.9% / 65.7%
o_proj      896x896         5.5%      20.0%      60.2%      240 of 896     3.4% / 23.6%
gate_proj  4864x896         5.4%      12.8%      35.9%      530 of 896     1.8% / 13.2%
up_proj    4864x896         4.8%      12.7%      38.0%      496 of 896     1.8% / 13.2%
down_proj   896x4864        2.4%       9.2%      33.2%      540 of 896     1.8% / 13.2%

It is far from it. The best rank-1 matrix captures 2 to 14% of an update’s squared size, and capturing 90% takes 60 of 128 directions for the key projection and about 500 of 896 for the MLP matrices. The updates have more structure than random matrices (rank 8 captures two to eight times as much as it would of noise), but they look like the widget’s second case far more than its first, as Biderman et al. found for full fine-tuning at larger scale. Yet a rank-1 LoRA came within about 0.01 of this fine-tune’s gain on the blog (one seed). Both results hold because they measure different things: the size of the update full fine-tuning happened to make, and the size of an update that suffices. AdamW moves every entry of every matrix at every step, by up to about the learning rate; where the gradient is mostly noise, those moves add up like a random walk, which is full-rank, so much of a full fine-tune’s update can be movement the loss barely depends on.

That can be tested: rebuild the fine-tuned model with each matrix’s update cut to its best rank-$k$ approximation, and measure it as before. truncated_update.py takes the SVD of every update once (svd), then rebuilds and measures the model at each rank:

print("full fine-tune's update cut to rank k in every block matrix (bits per byte)")
for k in ("pretrained", 0, 1, 8, 64, "all"):
    sd = dict(base)
    if k != "pretrained":
        sd.update(tuned)                                            # fine-tuned biases, norms, matrices
        if k != "all":
            for name, (U, S, Vh) in svd.items():
                sd[name] = (base[name].double() + (U[:, :k] * S[:k]) @ Vh[:k]).float()
    model.load_state_dict({n: v.to(DEVICE) for n, v in sd.items()})
    r = {n: bits_per_byte(*v) for n, v in evals.items()}
    print(f"  {str(k):>10s}: blog {r['blog']:.3f}, wiki {r['wiki']:.3f}", flush=True)

Full script: truncated_update.py

full fine-tune's update cut to rank k in every block matrix (bits per byte)
  pretrained: blog 1.163, wiki 0.981
           0: blog 1.162, wiki 0.981
           1: blog 1.122, wiki 0.998
           8: blog 1.103, wiki 1.010
          64: blog 1.078, wiki 1.026
         all: blog 1.068, wiki 1.047

The top directions carry the gain out of proportion to their size. Rank 8, 9 to 46% of each matrix’s squared update, recovers 63% of the blog gain (0.060 of 0.095); rank 64 recovers 89%. The rest costs more than it gains: the last step, from rank 64 to the full update, adds 0.010 bits per byte on the blog and costs 0.021 on WikiText. Most of the forgetting is in the top directions too, though (0.045 of the 0.066 by rank 64), and the forgetting per unit of gain rises with every step up in rank. The biases and normalization gains alone (rank 0) do nothing, and the full rebuild reproduces the fine-tune’s 1.068 and 1.047 even with the final normalization layer, which wasn’t saved, left pretrained. A truncated full fine-tune is still a worse low-rank update than one trained as low-rank: cut to rank 1 it reaches 1.122 on the blog, where trained rank-1 LoRAs reached 1.073 to 1.088. Truncation keeps each matrix’s largest direction, by size, not by effect on the loss, chosen one matrix at a time, and the random-walk part of the update inflates those sizes too; training picks all 168 directions together, against the loss. (The spectrum is one seed at step 400, and leaves out the biases and normalization gains the full fine-tune also changed. At rank 64 the 128-wide key and value matrices are half full-rank, so even noise would show 66% there.)

Merging changes nothing. After training, each $W$ was replaced by $W + \frac{\alpha}{r} BA$ and the LoRA layers removed. On a 512-token window the merged model’s logits differ from the unmerged one’s by at most $6.2 \times 10^{-4}$. The check runs in 32-bit floats, and that is rounding: $W x + \frac{\alpha}{r} B(Ax)$ and $(W + \frac{\alpha}{r} BA) x$ add the same numbers in a different order, and the differences build up over 24 blocks. The merged model is the trained model, at the base model’s cost.

It wasn’t faster on this laptop, an Apple M4 with 16 GB. A run took 25 to 35 minutes whatever the rank, a time that includes the six evaluations. In principle LoRA saves some compute as well as memory: the backward pass still goes through every frozen layer, to carry the gradient down to the LoRA matrices below, but it skips the frozen matrices’ weight gradients, about a third of a step’s arithmetic, or a quarter with the recomputation that gradient checkpointing adds, as here. Fine-Tuning, and What It Destroys’s full fine-tunes took about as long: at this size, on this laptop, the time goes to what both share (the forward pass, its recomputation, and the evaluations), not to the weight gradients LoRA skips.

Prefix tuning, prompt tuning and adapters

LoRA isn’t the only way to train a few parameters. Three others, older and still in use:

• Adapters (Houlsby et al., 2019): small bottleneck MLPs, down to 8 to 256 dimensions and back up, inserted after the attention and MLP sublayers, with everything else frozen. On GLUE they came within 0.4% of full fine-tuning, adding 3.6% parameters per task. Unlike LoRA they can’t be merged: they are extra layers in the forward pass, with extra latency.
• Prefix tuning (Li and Liang, 2021): learn a few vectors that are prepended to the keys and values at every layer, as if the model had read a prefix that no sequence of real tokens could produce. Training 0.1% of the parameters, it matched full fine-tuning on table-to-text generation and came slightly short on summarization.
• Prompt tuning (Lester et al., 2021): learn only a few embedding vectors at the input, a “soft prompt”. It is the smallest of all, and the paper found it closes the gap to full fine-tuning only at scale: weak on small models, competitive at around 10 billion parameters.

Prefix and prompt tuning also use up context: the learned vectors occupy positions a real prompt could use. LoRA became the default because it has none of these costs: no added latency once merged, no context used, and quality close to full fine-tuning on most tasks.

Serving many LoRAs

A model with a hundred fine-tunes as merged copies needs a hundred copies of the weights. As unmerged LoRAs it needs one copy and a hundred small pairs of matrices. A batch can then mix requests for different adapters: the base model’s matrix multiplication is shared by every request, and each request adds its own $BAx$. S-LoRA (Sheng et al., 2024) showed thousands of adapters served from one GPU this way, with adapters kept in CPU memory and loaded as requests arrive. vLLM and other servers support it, and vLLM Architecture measures what it costs from the serving side.

Where this leads

• Knowledge Distillation trains a small model to match a large one’s output distribution, on the same blog text and setup.
• Quantization from Scratch quantizes models, including the NF4 format QLoRA builds on.
• vLLM Architecture serves many LoRA fine-tunes from one copy of a model and measures the cost.

Takeaways

• A low-rank update is often enough for a fine-tune, so LoRA trains a rank-$r$ factorization $BA$ beside each frozen matrix: here rank 1, 0.11% of the parameters, learned the blog about as well as rank 8. Larger fine-tunes can need higher ranks, and outgrow LoRA.
• The savings are mostly memory: gradients and optimizer state exist only for $A$ and $B$, taking an 8B fine-tune from about 128 GB to 17 GB, or 6 GB with QLoRA’s 4-bit frozen weights. A step still runs the whole model forward and backward.
• $B$ starts at zero, so training starts from the pretrained model, and the trained update merges into $W$ with no change in output beyond rounding and no cost at inference.
• The scale $\alpha / r$ sets the step size. With $\alpha = 2r$ the learning rate has to fall as the rank rises: at $10^{-3}$ rank 64 got worse from the first checkpoint and rank 8 overshot, while rank 8 at $2 \times 10^{-4}$ was the best two-seed run.
• LoRA forgets as well as learns, and the forgetting keeps growing after the gain stops, so stop early.
• A full fine-tune’s own update here was not low-rank: rank 64 captured a third to two-thirds of each large matrix’s update, and rank 1 under a seventh. Yet its top 8 directions recovered 63% of its gain, and the high-rank remainder did more harm than good: a low-rank update that suffices and the update full fine-tuning makes are different things.
• LoRA doesn’t forget less by itself: here, at the same gain, full fine-tuning at $10^{-5}$ forgot less than every LoRA run, about 0.03 bits per byte against 0.045 to 0.055. The step size decides forgetting more than which weights move.
• Adapters add latency and prefix and prompt tuning use up context, costs a merged LoRA doesn’t have; unmerged LoRAs let one server hold thousands of fine-tunes over one copy of the base model.

References

Start with Hu et al.’s LoRA paper, then Biderman et al.’s LoRA Learns Less and Forgets Less for the careful comparison with full fine-tuning. QLoRA (Dettmers et al.) is the one to read before fine-tuning on a single GPU.

• Aghajanyan, Zettlemoyer and Gupta, Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (ACL, 2021): intrinsic dimension, about 200 parameters for 90% of full fine-tuning on MRPC.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR, 2022): the method itself, with low-rank updates, $B = 0$ initialization, merging.
• Dettmers, Pagnoni, Holtzman and Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS, 2023): NF4, double quantization, a 65B model on one 48 GB GPU.
• Biderman et al., LoRA Learns Less and Forgets Less (TMLR, 2024): LoRA against full fine-tuning, on both learning and forgetting.
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (2023): rsLoRA, scaling by $\alpha / \sqrt{r}$.
• Houlsby et al., Parameter-Efficient Transfer Learning for NLP (ICML, 2019): adapters, within 0.4% of full fine-tuning on GLUE.
• Li and Liang, Prefix-Tuning: Optimizing Continuous Prompts for Generation (ACL, 2021): prefix tuning, matching full fine-tuning on table-to-text with 0.1% of the parameters.
• Lester, Al-Rfou and Constant, The Power of Scale for Parameter-Efficient Prompt Tuning (EMNLP, 2021): prompt tuning, and how its gap to full fine-tuning closes with scale.
• Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters (MLSys, 2024): thousands of adapters served from one GPU.