Tag: adapters
All the articles with the tag "adapters".
Parameter-Efficient Fine-Tuning: LoRA, Prefix and Prompt Tuning
Updated:A low-rank update is often enough to fine-tune a model, so training only a low-rank part of it cuts trainable parameters and optimizer memory by orders of magnitude, and the result merges back into the weights for free. Intrinsic dimension, LoRA's shapes, initialization and scale, worked memory for an 8B model, QLoRA, prefix and prompt tuning and adapters, and serving many LoRAs at once. Then a hand-written LoRA on the fine-tune from Fine-Tuning, and What It Destroys at ranks 1, 8 and 64, two seeds each: what it learns and what it forgets next to full fine-tuning, how low-rank the full fine-tune's own update is, and a check that merging changes nothing.