Tag: rejection-sampling
All the articles with the tag "rejection-sampling".
Speculative Decoding, Derived: Draft Models, N-grams, Medusa and EAGLE
Updated:Decoding is memory-bound, so checking k guessed tokens costs about as much as generating one. A rejection-sampling rule makes the output exactly the target model's distribution whatever the guesser, proved and checked by sampling. The expected tokens per step derived and checked, the speedup with the draft's cost, and an interactive simulator; draft models, prompt lookup, Medusa heads and EAGLE; and acceptance measured on real models for two kinds of task.