Tag: inference-serving

All the articles with the tag "inference-serving".

• vLLM vs. SGLang vs. TensorRT-LLM: Comparing LLM Serving Architectures

  Updated:19 Oct, 2026

  vLLM, SGLang and TensorRT-LLM now share one design: continuous batching over a paged KV cache with prefix reuse, run from PyTorch with CUDA graphs. What differs is defaults, hardware and ecosystem. Measured: on multi-turn chat with room for the working set, the default prefix caches reused 92.0% of prompt tokens in SGLang, 90.9% in vLLM and 92.0% in TensorRT-LLM; with a smaller cache, TensorRT-LLM's fell furthest. The rest is each engine's source, read at its latest release.

• TensorRT-LLM: Compiling Inference Engines Ahead of Time

  Updated:16 Oct, 2026

  TensorRT-LLM began by compiling each model into an engine: operations fused into fewer kernels, and every kernel timed and chosen for the GPU and the shapes it would serve. Both steps measured on a laptop GPU: fusion removed 26% of the operations per token and 7% of prefill time; the best version of one kernel depended on the shape, and the worst cost twice as much. Then why NVIDIA moved the project onto a PyTorch runtime, and what it keeps of the compiled design.