Tag: inference-serving
All the articles with the tag "inference-serving".
The KV Cache: How Transformers Avoid Recomputing the Past
Updated:Generating text recomputes every earlier token's keys and values unless they are kept. The KV cache keeps them, and it is exact: in Qwen2.5-1.5B-Instruct the next token matched at 32 of 32 steps. It turned a step at 4,096 tokens from 4.7 s into 65 ms on a laptop. Then what it costs: 28 KiB per token for Qwen, 800 KiB for Llama 2 13B, why decoding is limited by memory reads, how batch size trades latency for throughput, and how GQA, MLA and cross-layer sharing shrink the cache.
Paged Attention: Managing the KV Cache Like Virtual Memory
Updated:A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.
SGLang and RadixAttention: Sharing the KV Cache Across Requests
Updated:Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.
vLLM Architecture: Continuous Batching and Serving at Scale
Updated:A serving engine is a loop that decides, every forward pass, which requests' tokens go into it. What a pass costs on a real GPU, measured; static batching against continuous batching; vLLM V1's scheduler, where prefill and decode are one rule under a token budget, written out and run on ShareGPT traffic with the measured costs, with paged memory and preemption; the processes vLLM runs; tensor parallelism's communication, worked for a 70B model; and serving many LoRA fine-tunes from one model, measured.