Tag: fine-tuning

All the articles with the tag "fine-tuning".

• Parameter-Efficient Fine-Tuning: LoRA, Prefix and Prompt Tuning

  Updated:19 Oct, 2026

  A low-rank update is often enough to fine-tune a model, so training only a low-rank part of it cuts trainable parameters and optimizer memory by orders of magnitude, and the result merges back into the weights for free. Intrinsic dimension, LoRA's shapes, initialization and scale, worked memory for an 8B model, QLoRA, prefix and prompt tuning and adapters, and serving many LoRAs at once. Then a hand-written LoRA on the fine-tune from Fine-Tuning, and What It Destroys at ranks 1, 8 and 64, two seeds each: what it learns and what it forgets next to full fine-tuning, how low-rank the full fine-tune's own update is, and a check that merging changes nothing.

• Fine-Tuning, and What It Destroys

  Updated:16 Oct, 2026

  Fine-tuning moves a pretrained model a short distance in weight space, and how far it moves decides what it gains and what it forgets. Continued pretraining vs supervised fine-tuning, the SFT loss with prompt masking worked on a real chat template, learning rates and memory. Then Qwen2.5-0.5B fine-tuned on this blog, two seeds per setting, with what it learns and what it forgets measured side by side; replay as the remedy, EWC derived, and when fine-tuning is the wrong tool.