Tag: structured-generation

All the articles with the tag "structured-generation".

• SGLang and RadixAttention: Sharing the KV Cache Across Requests

  Updated:19 Oct, 2026

  Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.