Tag: prefix-caching

All the articles with the tag "prefix-caching".

• SGLang and RadixAttention: Sharing the KV Cache Across Requests

  Updated:19 Oct, 2026

  Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.

• vLLM vs. SGLang vs. TensorRT-LLM: Comparing LLM Serving Architectures

  Updated:19 Oct, 2026

  vLLM, SGLang and TensorRT-LLM now share one design: continuous batching over a paged KV cache with prefix reuse, run from PyTorch with CUDA graphs. What differs is defaults, hardware and ecosystem. Measured: on multi-turn chat with room for the working set, the default prefix caches reused 92.0% of prompt tokens in SGLang, 90.9% in vLLM and 92.0% in TensorRT-LLM; with a smaller cache, TensorRT-LLM's fell furthest. The rest is each engine's source, read at its latest release.