vLLM vs. SGLang vs. TensorRT-LLM: Comparing LLM Serving Architectures

vLLM Architecture, SGLang and RadixAttention and TensorRT-LLM followed three engines that started from three different ideas: a paged KV cache with a step-by-step scheduler, a radix tree of every cached prefix, and a model compiled ahead of time into an engine. Two years on, the three have converged. Each batches continuously over a paged KV cache, reuses cached prefixes across requests, runs the model from PyTorch with CUDA graphs, and serves an OpenAI-compatible API.

What still separates them is smaller and more practical: what each turns on by default, which hardware each runs on, and who maintains it. This post compares them on those terms, reading each engine’s source at its latest release (vLLM v0.31.0, SGLang v0.5.21, TensorRT-LLM v1.2.1, the latest stable), and measures the one architectural difference that remains, how finely each engine matches a cached prefix.

Table of contents

What the three share

Each piece of the shared design has a post of its own. Continuous batching (TensorRT-LLM calls it in-flight batching) admits and retires requests at every step, as vLLM Architecture followed. The KV cache lives in fixed-size blocks, as in Paged Attention. Cached prefixes are shared across requests, and all three evict roughly least recently used first (TensorRT-LLM by priority, then recency; see Limits). Each captures CUDA graphs, recordings of a decoding step’s kernel launches that replay with one launch, and each ships an OpenAI-compatible server (on ports 8000, 30000 and 8000 by default), so a client written for one talks to the others by changing a URL.

The compilation story converged too, but less than it looks. All three use the term piecewise CUDA graph, a model’s graph cut into pieces, each captured separately, so the parts that cannot be captured run between them (in vLLM, attention). The defaults differ:

So “all three are PyTorch runtimes” holds for the defaults; only vLLM fuses kernels with Inductor out of the box, and the compiled engine that set TensorRT-LLM apart survives only in its last stable release. TensorRT-LLM measures what that engine’s build step did.

Batching and scheduling

The schedulers differ in their defaults and in what they do when memory runs out:

vLLM’s V1 engine only recomputes a preempted request; the swap to CPU memory of its earlier engine is gone. SGLang’s cache-aware policy, longest prefix match first, is opt-in: SGLang and RadixAttention measured it reusing 58 to 70% of prompt tokens on a shared-prefix burst against 8 to 9% in arrival order, while the slowest early arrival waited about twice as long.

Prefix reuse, measured

All three engines keep cached prefixes for later requests, engine-wide, but match them at different granularity:

• vLLM names each full 16-token block by a hash of its tokens and its parent’s hash, and looks blocks up in a table. A match covers whole blocks only.
• SGLang keeps a radix tree of tokens with one token per page, so a match can end anywhere. On attention backends that need bigger pages (64 tokens with the default backend on Blackwell GPUs) it matches by pages instead.
• TensorRT-LLM stores 32-token blocks “in a radix search tree as soon as they are filled”, and has partial reuse on by default: when only some of a block’s tokens match, “a new block is created and the matched tokens are copied into the new block”. A request’s last, partly filled block is stored too. Its match therefore also ends at the token.

A block cache with partial reuse, as a sketch of TensorRT-LLM’s default (whole blocks by hash, then the next stored block compared token by token):

    def match(self, tokens):
        self.clock += 1
        n = 0
        for depth, (parent, h, run) in enumerate(self._chain(tokens)):
            if len(run) == self.block and h in self.blocks:      # a whole block, by hash
                self.blocks[h] = (self.clock, -depth)
                n += self.block
                continue
            best, hit = 0, None                                  # partial: compare token by token
            for child, stored in self.children.get(parent, {}).items():
                k = 0
                while k < min(len(run), len(stored)) and run[k] == stored[k]:
                    k += 1
                if k > best:
                    best, hit = k, child
            if hit is not None:
                self.blocks[hit] = (self.clock, -depth)
                n += best
            break
        return n

Full script: partial.py

The workload is the replay from SGLang and RadixAttention: 2,161 requests from 429 multi-turn ShareGPT conversations, interleaved as they would arrive, each prompt the whole conversation so far. Requests run one at a time: each is matched against the cache, then its prompt and answer are cached before the next arrives, and the cache’s capacity ignores the memory running requests hold. Real eviction differs (TensorRT-LLM, for one, evicts by priority); here all three evict the least recently used first and free leaves before their ancestors, a whole leaf run of tokens in the radix tree and one block in the block caches. So the caches differ in how they match prefixes and in the unit they store and evict, not in eviction policy. Then whole blocks alone, at four sizes:

Full script: reuse.py

2161 requests from 429 multi-turn conversations; 1,583,998 prompt tokens, 733 per request
prompt tokens reused, each engine's default match
cache size     SGLang (any prefix)   vLLM (16-token blocks)   TensorRT-LLM (32, partial reuse)
    65,536             80.5%                 79.4%                    77.5%
   262,144             92.0%                 90.9%                    92.0%
 unlimited             92.0%                 90.9%                    92.0%
whole blocks only, by block size (tokens)
cache size           16      32      64     128
    65,536        79.4%   77.7%   76.6%   74.2%
   262,144        90.9%   89.6%   87.8%   84.2%
 unlimited        90.9%   89.6%   87.8%   84.2%

With room for the working set, SGLang and TensorRT-LLM both reuse 92.0% of prompt tokens and vLLM 90.9%: the 1.1 points are the partial block at the end of each match, which vLLM recomputes. Whole blocks alone lose more as they grow, 84.2% at 128 tokens; with longer prompts the loss should be a smaller share, since it is at most one block per request (these prompts average 733 tokens).

With a 65,536-token cache the order changes: TensorRT-LLM’s reuse falls to 77.5%, below vLLM’s 79.4%. Its 32-token blocks are a coarser unit of memory: a conversation’s partly filled last block takes a whole block, which likely means fewer prefixes fit (the replay does not separate this out). The cache’s size matters most of all: every engine loses 11.5 to 14.5 points going from enough memory to 65,536 tokens. On this workload, how an engine matches prefixes is not what separates the three; how much memory it leaves for the cache, and whether the scheduler orders requests to use it, matter more.

Structured outputs

All three now hand grammars to the same few libraries, the speed of which comes from computing the mask of allowed tokens quickly:

vLLM’s auto makes “opinionated choices based on request contents and what the backend libraries currently support, so the behavior is subject to change in each release”; it uses Outlines only for some Mistral tokenizers and schemas guidance cannot handle. vLLM added xgrammar as its default in v0.6.5 (December 2024), and its blog the next month reported “up to 5x improvement in time per output token (TPOT) under load”. SGLang made xgrammar its default in a v0.4.4 patch release (March 2025); in v0.4 it was opt-in.

Speculative decoding and quantization

Speculative Decoding, Derived explains the methods. The lists at each engine’s latest release:

vLLM’s list is the longest, and it is the only one of the three with Medusa on its default backend.

For quantization, all three run FP8, 4-bit weight formats (GPTQ, AWQ) and NVIDIA’s 4-bit floating-point NVFP4, and all three load checkpoints from NVIDIA’s Model Optimizer, whose README describes them as “ready for deployment in downstream inference frameworks like SGLang, TensorRT-LLM, TensorRT, or vLLM”. The differences are at the edges: SGLang loads GGUF files in-tree and vLLM through a separate plugin; INT8 with quantized activations (SmoothQuant’s W8A8) is in vLLM’s and SGLang’s lists and gone from TensorRT-LLM’s recipes, which keep INT8 for weights only. Quantization from Scratch derives the formats.

Hardware

This is the clearest difference, and the one most often misstated. Each engine runs some platforms from its own repository and others through separate plugins or engines:

vLLM lists the most platforms, most of them maintained as plugins under the vLLM project; SGLang covers the major ones and a few vLLM does not; TensorRT-LLM runs only on NVIDIA GPUs under Linux. A plugin can trail the core engine’s features, so for anything other than NVIDIA and AMD the question is not only “is it supported” but “which features work there”.

Serving many replicas

A single server is the same story in all three. Across replicas, the problem each solves is routing a request to the replica that already holds its prefix:

• SGLang introduced a cache-aware load balancer in v0.4 (December 2024), which keeps “an approximate radix tree of the actual radix tree on the workers”; LMSYS reported up to 1.9 times the throughput. Now called the SGLang Model Gateway, its default policy, cache_aware, “Combines cache locality with load balancing”, and it can route to any OpenAI-compatible server.
• vLLM points to two projects. The vLLM Production Stack, a Kubernetes deployment, lists prefix-aware routing as work in progress in its README. llm-d, a Cloud Native Computing Foundation sandbox project with vLLM as its main engine, “reads vLLM’s KV-cache events and routes each request to the replica that already holds its prefix”.
• TensorRT-LLM’s trtllm-serve supports disaggregated serving, prompts and decoding on separate GPU pools (Serving Metrics and Prefill–Decode Disaggregation).

NVIDIA’s own products now span the three. Dynamo, its orchestration layer, “doesn’t replace SGLang, TensorRT-LLM, or vLLM” and runs all three. NIM, its packaged containers, used to choose a backend by a priority order (“TensorRT-LLM > vLLM > SGLang”, now listed as no longer supported); NIM 2.0 adopts “a one-container, one-backend philosophy”, with vLLM as the backend for most models and SGLang for a few.

Who maintains them

A hosted project is governed by the foundation; an ecosystem listing is a directory entry. The user lists are each project’s own.

Benchmarks

No engine’s own benchmarks compare like with like across the three: each team tunes its own engine and reports its own numbers. The one regular third-party comparison is SemiAnalysis’s InferenceX (formerly InferenceMAX), which describes itself as an “independent, vendor neutral, reproducible benchmark” and reruns vLLM, SGLang and TensorRT-LLM on NVIDIA and AMD GPUs as the engines update. Its results change with each release, which is the main lesson: a ranking from one month’s versions on one GPU does not carry over to the next.

Choosing

• Hardware decides first. Anything other than NVIDIA rules out TensorRT-LLM; for platforms beyond NVIDIA and AMD, check which features the plugin supports.
• Then the workload. SGLang’s longest-prefix-first scheduler and its cache-aware router are built for traffic that shares prefixes, such as agents or few-shot prompts with a common context, at a cost in fairness. A specific speculative decoding method, such as Medusa, can decide it too.
• Then measure. On NVIDIA GPUs with an ordinary workload, the three are close enough that the answer depends on the model, the GPU and the versions; run the target workload on each.

Limits

• The defaults are a snapshot, read from source at three releases in October 2026. They change often.
• The reuse measurement isolates one difference. The three caches also differ in eviction (TensorRT-LLM evicts only leaf blocks, by priority), SGLang’s page size depends on its attention backend, and the replay runs requests one at a time with no memory held by running requests.
• No throughput numbers. Comparing the engines’ speed needs their own kernels on the same GPUs; none ran here.

Where this leads

• Serving Metrics and Prefill–Decode Disaggregation defines what a benchmark of these engines should measure.
• Running LLMs on a Laptop follows llama.cpp, built for the machines these engines do not target.

Takeaways

• vLLM, SGLang and TensorRT-LLM share one design: continuous batching over a paged KV cache, prefix reuse across requests, CUDA graphs and an OpenAI-compatible server. Only vLLM compiles with Inductor by default, and TensorRT-LLM’s stable release still carries its old engine backend.
• On multi-turn chat with room for the working set, their default prefix caches reused 92.0% (SGLang, any prefix), 90.9% (vLLM, whole 16-token blocks) and 92.0% (TensorRT-LLM, 32-token blocks with partial reuse) of prompt tokens. With a 65,536-token cache, 77.5 to 80.5%: cache size and scheduling order matter more than match granularity.
• Defaults differ: chunked prefill is off in TensorRT-LLM, SGLang’s cache-aware order is opt-in, and TensorRT-LLM has no grammar backend until one is set.
• Hardware separates them most: vLLM lists the most platforms, largely through plugins; SGLang the major ones; TensorRT-LLM NVIDIA only.
• Ecosystems differ: vLLM is a PyTorch Foundation project, SGLang is hosted by LMSYS, TensorRT-LLM is NVIDIA’s. Rankings of their speed are dated by the versions they measured.

References

Start with each engine’s configuration source at the versions above, where the defaults live, then the InferenceX about page for how a fair comparison is run.

• vLLM v0.31.0 source, vllm/config/: the scheduler, cache, compilation and structured-output defaults.
• SGLang v0.5.21 source, python/sglang/srt/arg_groups/: server arguments and their defaults; docs/docs/hardware-platforms/ for platforms.
• TensorRT-LLM v1.2.1 source, tensorrt_llm/llmapi/llm_args.py: the PyTorch runtime’s defaults.
• vLLM Blog, Structured Decoding in vLLM: a Gentle Introduction (2025): the xgrammar integration and the 5x figure.
• LMSYS, SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs (2024): the cache-aware load balancer.
• vLLM, llm-d and Production Stack: vLLM’s multi-replica routing.
• NVIDIA, NIM LLM 2.0 release notes and Model Profiles and Selection: NIM 2.0’s move to one backend, and the backend priority it dropped.
• NVIDIA, KV Cache System: TensorRT-LLM’s radix search tree and partial reuse.
• SGLang, SGLang Model Gateway: the cache-aware router’s policies.
• NVIDIA, Dynamo and Model Optimizer: the orchestration layer over all three, and the quantization library all three load.
• PyTorch Foundation, PyTorch Foundation Welcomes vLLM as a Hosted Project (2025), and SGLang Team, SGLang Joins PyTorch Ecosystem (2025): the governance and the named users.
• vLLM Blog, vLLM 2024 Wrapped & 2025 Vision (2025): vLLM’s named users.
• SemiAnalysis, InferenceX: the third-party benchmark of all three engines.