Tag: paged-attention
All the articles with the tag "paged-attention".
Paged Attention: Managing the KV Cache Like Virtual Memory
Updated:A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.