vLLM Architecture: Continuous Batching and Serving at Scale

Paged Attention stored the KV cache in blocks, so that memory holds real tokens instead of reservations. That is half of a serving engine. The other half is the loop that runs the model: at every forward pass it decides which requests take part, how many of their tokens, and what to do when memory runs out. vLLM is a paged KV cache plus that loop, and the loop decides most of what a user notices: how long until the first token, how smooth the stream is, how many users one GPU can carry. This post measures what one pass costs, builds the scheduler, runs it on real request lengths, and then follows vLLM’s processes, its multi-GPU modes and its multi-LoRA serving.

Table of contents

What one step costs

Every scheduling decision weighs the same quantity: the time of one forward pass, given which tokens are in it. A decode token is one new token of a running answer, which attends over that request’s cached keys and values; a prefill token is a prompt token, processed many at a time. The KV Cache predicted that decoding is limited by reading the weights, so a pass over many sequences should cost little more than a pass over one. Measured on Qwen2.5-1.5B, on the Apple laptop GPU of How a GPU Runs Deep Learning (lm.py loads the model as in Decoding), one decode step:

@torch.no_grad()
def decode_step(B, C):
    ids = torch.tensor([blog[i * C:(i + 1) * C] for i in range(B)], device=DEVICE)
    cache = DynamicCache()
    model.model(ids, past_key_values=cache, use_cache=True)          # fill the cache, no logits
    def step():
        model(ids[:, -1:], past_key_values=cache, use_cache=True)
        cache.crop(-1)                                               # keep the context fixed
    return timed(step)

and, after timing prompts the same way, the cost model fitted to all of them:

X = np.array([[1, n, c, q] for n, c, q, _ in rows], float)
y = np.array([t for *_, t in rows])
coef = np.linalg.lstsq(X / y[:, None], np.ones(len(y)), rcond=None)[0]   # fit relative error
err = np.abs(X @ coef - y) / y
print(f"fit: {coef[0] * 1e3:.1f} ms + {coef[1] * 1e6:.0f} us per new token + {coef[2] * 1e9:.0f} ns per cached "
      f"token read + {coef[3] * 1e12:.1f} ps x T^2; error over {len(y)} steps: mean {err.mean():.1%}, max {err.max():.1%}")
np.save("cost.npy", coef)

Full script: steps.py, with the prefill timing and the printing. It takes about 5 minutes on this laptop’s Apple M4.

decode, ms per step (tokens/s)
   B                C = 128             C = 512            C = 2048
   1         46.0 (   22/s)      42.2 (   24/s)      55.5 (   18/s)
   4         49.9 (   80/s)      49.5 (   81/s)      66.1 (   61/s)
  16         74.0 (  216/s)      84.3 (  190/s)     131.5 (  122/s)
  64        127.7 (  501/s)     161.5 (  396/s)                    
 128        196.2 (  652/s)     280.3 (  457/s)                    
prefill, one sequence
  T =   16:    53.8 ms (  297 tokens/s)
  T =   64:    80.1 ms (  799 tokens/s)
  T =  128:   136.3 ms (  939 tokens/s)
  T =  256:   245.5 ms ( 1043 tokens/s)
  T =  512:   480.7 ms ( 1065 tokens/s)
  T = 1024:   990.9 ms ( 1033 tokens/s)
  T = 2048:  2082.3 ms (  984 tokens/s)
  T = 4096:  4303.0 ms (  952 tokens/s)
fit: 44.2 ms + 778 us per new token + 2334 ns per cached token read + 76514.7 ps x T^2; error over 21 steps: mean 6.7%, max 17.7%

Decoding one sequence took 42 ms, and decoding 16 took 84: sixteen times the tokens for twice the time, because the pass reads all 2.9 GiB of weights whatever the batch, about 34 ms at the 92 GB/s measured there, most of the 42. The fit puts each token’s arithmetic at 0.78 ms (3.1 GFLOP, about 4 TFLOPS), so from about 40 sequences the arithmetic outweighs the weight read, and throughput rises more slowly up to 128. A prompt’s tokens go through the model together, so one prompt of 256 tokens already runs at about 1,000 tokens a second, the arithmetic ceiling, and longer prompts add their attention over themselves. The fitted model, 44 ms per pass plus costs per token, per cached token read and for a prompt’s self-attention (fitted to minimize relative error), is within 7% of these 21 measurements on average and 18% at worst. The simulations below use it.

The cost per cached token, 2.3 µs, is about seven times what reading a token’s 28 KiB of keys and values at 92 GB/s would take. Part of the excess is how this measurement runs, not attention itself: Hugging Face’s cache copies the whole cache into a new tensor at every step (torch.cat), which a paged engine doesn’t, so the simulations over-charge long contexts.

The lesson for a scheduler: a pass should hold as many decoding requests as possible, and a long prompt takes a long time whatever else is in the pass.

Static batching

The simplest way to batch, and the one a training loop uses, is static: collect a batch of requests, run them until every one has finished, then take the next batch. For generation it fails twice. A request that finishes early holds its slot, idle, until the longest answer in the batch ends; and a request that arrives just after a batch starts waits for the whole batch before it is even read.

Continuous batching decides the batch again at every step: a finished request leaves, and a waiting one joins, at the next forward pass. Yu et al. (2022) introduced it in Orca as iteration-level scheduling and measured 36.9 times FasterTransformer’s throughput at the same latency. Joining a running batch is only cheap if the newcomer’s cache needs no large contiguous reservation, which is what paged attention’s blocks provide; a request takes blocks one at a time as its answer grows.

The scheduler, as one rule

vLLM’s original engine, by default, ran two kinds of step: a step that prefilled newly admitted prompts, and a step that decoded every running request (chunked prefill came later, as an option). vLLM V1, the only engine since version 0.11 (October 2025), replaced both with one rule. Each request records how many of its tokens have their keys and values computed, and how many tokens it has (prompt plus answer so far); a step gives tokens to requests so that the first number catches up with the second, within a token budget. The scheduler’s own comment puts it this way: “There’s no ‘decoding phase’ nor ‘prefill phase’ in the scheduler.” A decoding request is one that is a single token behind; a new request is a whole prompt behind; a request evicted from memory is its prompt and answer behind. Running requests are served first, then waiting ones, in arrival order, with prompts cut into chunks to fit what is left of the budget (chunked prefill, after Agrawal et al.’s Sarathi).

The simulation below implements that rule, the older two-step one, and static batching, with a pool of 16-token blocks and preemption when the pool runs out. The two-step rule (prefill-first below) caps a prefill step at 4,096 prompt tokens. Requests arrive at random times, at a fixed average rate, with the prompt and answer lengths of the ShareGPT requests in Paged Attention; each step takes the time the fitted model gives for its tokens. It is a simulation of the scheduling, not a running engine: the costs are this laptop’s, measured above, and the GPU work itself isn’t done. It assumes the costs of decode and prefill tokens in one step add, which the measurements didn’t test, and gives static batching unlimited memory. Each run is 2,000 requests; at these rates, which the laptop can serve, the continuous schedulers reach a steady state.

A request records its state in two numbers:

class Req:
    def __init__(self, i, t, prompt, answer):
        self.i, self.arrive, self.prompt, self.answer = i, t, prompt, answer
        self.done = 0                         # tokens whose keys and values are computed
        self.out = []                         # times its answer tokens came out
        self.blocks = 0

    def known(self):                          # prompt plus answer so far
        return self.prompt + len(self.out)

    def decoding(self):                       # all but the newest token computed: feed it, get the next
        return self.out and self.done == self.known() - 1

and a V1 step, for the continuous schedulers, is the rule above: running requests’ next tokens first, preempting the newest when the pool runs out, then unfinished prompts, then waiting requests, under the budget:

                for r in list(running):                   # running requests' next token first
                    if r in running and r.decoding():
                        while r in running and not grow(r, r.done + 1):
                            victim = running.pop()        # out of memory: preempt the newest
                            free += victim.blocks; victim.blocks = victim.done = 0
                            waiting.insert(0, victim); preempted += 1
                        if r in running:
                            decodes.append(r)
                left = budget - len(decodes)
                for r in running:                         # prompts still being prefilled, in chunks
                    if not r.decoding() and left > 0:
                        k = min(r.known() - r.done, left)
                        if grow(r, r.done + k):
                            prefills.append((r, k)); left -= k
            if mode == "chunked" or not decodes:          # admit waiting requests while there is room
                while waiting and left > 0 and len(running) < max_seqs:
                    r = waiting[0]
                    k = r.known() - r.done                # after a preemption: prompt and answer so far
                    if mode == "chunked":
                        k = min(k, left)
                    elif k > left:
                        break
                    if not grow(r, r.done + k):
                        break
                    waiting.pop(0); running.append(r)
                    prefills.append((r, k)); left -= k

Full script: engine.py, with static batching, the cost of a step, the metrics and the runs below.

                          answer tok/s  TTFT p50  TTFT p99  time/token  p99 gap  longest gap
0.5 requests a second
  static, 16                     64   1058.4 s  2302.7 s      69 ms   0.08 s     0.08 s
  static, 128                    91     86.5 s   534.0 s     163 ms   0.28 s     0.28 s
  prefill-first                 101      0.2 s     1.0 s      63 ms   0.31 s     2.41 s
  chunked, 2048                 101      0.2 s     1.0 s      61 ms   0.29 s     1.30 s
  chunked, 512                  101      0.2 s     1.1 s      61 ms   0.31 s     0.50 s
1.0 requests a second
  static, 16                     65   1963.9 s  4189.7 s      69 ms   0.08 s     0.08 s
  static, 128                    98    740.3 s  2099.2 s     232 ms   0.28 s     0.28 s
  prefill-first                 199      0.2 s     1.3 s      99 ms   0.71 s     2.62 s
  chunked, 2048                 199      0.3 s     1.3 s      91 ms   0.58 s     1.55 s
  chunked, 512                  199      0.3 s     1.4 s      91 ms   0.48 s     0.52 s
1.5 requests a second
  static, 16                     66   2280.7 s  4800.7 s      69 ms   0.08 s     0.08 s
  static, 128                    99   1051.2 s  2689.6 s     235 ms   0.28 s     0.28 s
  prefill-first                 289      0.3 s     1.6 s     219 ms   1.45 s     4.19 s
  chunked, 2048                 291      0.4 s     1.9 s     177 ms   0.80 s     1.80 s
  chunked, 512                  291      0.4 s     2.1 s     177 ms   0.52 s     0.55 s
chunked, 512, 1.0 request a second, smaller KV pools
  4096 blocks (65,536 tokens):  199 tok/s, TTFT p99    1.4 s, longest gap  0.52 s,    0 preemptions
   512 blocks ( 8,192 tokens):  181 tok/s, TTFT p99  199.9 s, longest gap 28.62 s, 2004 preemptions
   256 blocks ( 4,096 tokens):  129 tok/s, TTFT p99 1114.8 s, longest gap 28.15 s, 2225 preemptions
   128 blocks ( 2,048 tokens):   79 tok/s, TTFT p99 3085.6 s, longest gap 42.55 s, 2190 preemptions

Three results.

• Static batching can’t keep up. Batches of 16 top out near 65 tokens a second and batches of 128 near 99, both less than even 0.5 requests a second ask for (about 103), so the queue only grows: at 0.5, the median request waited 18 minutes for its first token with batches of 16 and 1.4 minutes with batches of 128, longer at higher rates and the longer the run. Steady gaps between tokens are the one thing it does well, since a batch never admits anyone. Continuous batching carried all the traffic up to 1.5 requests a second; at 1.0 and 1.5 that was 2 to 3 times what static batches of 128 managed and 3 to 4.4 times batches of 16, with the median first token at 0.2 to 0.4 s. The Anyscale benchmark of 2023 found the same direction at data-center scale: on OPT-13B and one A100, continuous batching alone gave about 8 times the throughput of static batching, and with vLLM’s memory management 23 times.
• Whole prompts stall everyone else. With prefill-first scheduling, new prompts get steps of their own, and every running answer waits for them: at 1.5 requests a second some reader waited 4.2 s for a next token, and 1% of all gaps were longer than 1.45 s. Agrawal et al. (2024) call these generation stalls. A 512-token budget caps any step at 512 tokens of work, and the longest gap fell to 0.55 s, at slightly higher throughput. Serving Metrics and Prefill–Decode Disaggregation found the same in a model of an H100, and why the chunk should be sized to the latency target.
• The budget is a trade. A smaller budget spreads each prompt over more steps, so first tokens come later: the 99th-percentile first token went from 1.9 s (2,048) to 2.1 s (512) at 1.5 requests a second. vLLM’s default budget (as of version 0.31) depends on the hardware and entry point, from 2,048 for its OpenAI-compatible server on most GPUs to 16,384 on the largest; its documentation recommends smaller values for smoother streams and more than 8,192 for throughput.

When memory runs out

Admitting requests by what they hold now means the pool can run dry as their answers grow. The simulation’s last rows shrink the pool at 1.0 request a second. With 4,096 blocks, nothing is ever preempted. With 512 blocks (8,192 tokens), the scheduler preempted 2,004 times over the 2,000 requests, throughput fell from 199 to 181 tokens a second, and some reader waited 29 s between two tokens while a preempted request waited to come back; with 128 blocks, throughput fell to 79.

vLLM V1 preempts the way the simulation does: the most recently admitted running request gives up its blocks (by default; a priority policy picks the lowest priority instead), and goes to the front of the waiting queue with nothing computed, to recompute its prompt and answer so far when memory frees up. The original engine could also swap a preempted request’s blocks to CPU memory; V1 dropped that path, and its documentation says only that “recomputation has lower overhead in the V1 architecture”. Preemption is a sign of too little memory for the load, and the fix is upstream: fewer concurrent requests (max_num_seqs), a smaller cache per token (KV Cache Compression), or more GPUs.

The same rule covers two more cases, since each only changes how far behind a request is. A cached prefix, found by block hashes (Paged Attention) and on by default in V1, puts a new request less far behind. Speculative decoding (Speculative Decoding, Derived) puts a decoding request several tokens behind, the drafted ones, which one pass then verifies.

The processes

The scheduler has to decide, launch and collect a step in a few milliseconds, every few milliseconds, and the rest of a server’s work must not get in its way. V1 splits vLLM into processes (its architecture documentation):

• An API server accepts HTTP requests, tokenizes prompts, and detokenizes and streams the output (Streaming Tokens to the User).
• An engine core runs a busy loop: schedule a step, have the workers execute it, update every request. It talks to the API server over ZeroMQ sockets, so tokenizing a long prompt or streaming text never delays a step.
• One worker process per GPU loads its part of the model and runs it. An engine core drives (tensor parallel size) times (pipeline parallel size) workers; with data parallelism there is one engine core per replica. With four GPUs in tensor parallel, a server is six processes.

Two techniques keep the CPU side cheap. A persistent batch keeps the model’s input tensors between steps and applies only the changes (who joined, who left), instead of rebuilding them. And CUDA graphs record a decode step’s sequence of GPU kernels once and replay it with one launch, which NVIDIA describes as “a mechanism to launch multiple GPU operations through a single CPU operation, and hence reduce overheads”. A decode step is hundreds of kernels (this post’s model has 28 layers of a dozen or more operations each), many of them short, and each launch costs CPU time. vLLM captures full graphs for decode-only batches and piecewise graphs for steps with prefill, built with torch.compile. Together with the process split and the unified scheduler, the V1 rewrite as a whole reported up to 1.7 times the throughput of the old engine.

Many GPUs

When a model doesn’t fit on one GPU, or one GPU is too slow, vLLM splits the work four ways, which can be combined:

• Tensor parallelism (--tensor-parallel-size) splits every layer’s matrices across GPUs, as Megatron-LM does: the attention heads and the MLP’s first matrix by columns, the output projections by rows, so each GPU computes a slice and a layer needs “only two all-reduces in the forward path”, one after attention and one after the MLP.
• Pipeline parallelism (--pipeline-parallel-size) gives each GPU a range of layers; activations pass from one to the next.
• Data parallelism (--data-parallel-size) runs whole replicas, each with its own scheduler.
• Expert parallelism (--enable-expert-parallel) spreads a mixture-of-experts model’s experts over GPUs (Mixture of Experts).

What tensor parallelism costs is communication at every layer. For Llama-3-70B on eight H100s:

# Tensor parallelism, worked for Llama-3-70B on eight H100s: each GPU holds
# an eighth of every layer, and each layer ends in two all-reduces (after
# attention's output projection and after the MLP; Megatron-LM) of the
# activations, B tokens x 8,192 numbers in 16 bits. A ring all-reduce
# sends 2(N-1)/N times the buffer from each GPU (nccl-tests). Compared with
# the time to read each GPU's share of the weights, which sets a decode
# step's floor. NVIDIA's figures: HBM 3.35 TB/s; NVLink 900 GB/s both ways
# (450 each way); PCIe Gen5 x16 64 GB/s each way.
D, L, H, KV, HD, F, V = 8192, 80, 64, 8, 128, 28672, 128256
params = L * (D * H * HD * 2 + D * KV * HD * 2 + 3 * D * F) + 2 * V * D
N, HBM = 8, 3.35e12
weights_ms = params * 2 / N / HBM * 1e3
print(f"Llama-3-70B: {params / 1e9:.1f}B parameters, {params * 2 / 1e9:.0f} GB in 16 bits; "
      f"{params * 2 / N / 1e9:.1f} GB per GPU, read in {weights_ms:.1f} ms")
print("decode batch   all-reduce bytes per GPU per step   over NVLink   over PCIe Gen5   (weights: "
      f"{weights_ms:.1f} ms)")
for B in (1, 16, 64, 256):
    sent = L * 2 * B * D * 2 * 2 * (N - 1) / N
    print(f"{B:8d}       {sent / 1e6:12.1f} MB                    {sent / 450e9 * 1e3:6.2f} ms     "
          f"{sent / 64e9 * 1e3:6.2f} ms")

Llama-3-70B: 70.6B parameters, 141 GB in 16 bits; 17.6 GB per GPU, read in 5.3 ms
decode batch   all-reduce bytes per GPU per step   over NVLink   over PCIe Gen5   (weights: 5.3 ms)
       1                4.6 MB                      0.01 ms       0.07 ms
      16               73.4 MB                      0.16 ms       1.15 ms
      64              293.6 MB                      0.65 ms       4.59 ms
     256             1174.4 MB                      2.61 ms      18.35 ms

Full script: tp.py, shown whole above.

Each GPU reads its 17.6 GB of weights in 5.3 ms per decode step, and at batch 64 the all-reduces move 294 MB per GPU: 0.65 ms over NVLink, an eighth of the weight-reading time that sets the step’s floor, but 4.6 ms over PCIe, nearly doubling it (and that assumes each GPU gets a full PCIe link, which eight GPUs sharing a machine’s switches don’t). That is why tensor parallelism stays inside one machine’s NVLink domain, and pipeline parallelism, which sends one activation per layer range rather than two per layer, is what spans machines. (Real all-reduces also pay a latency per call, 160 calls per step here, which matters most at small batches.)

Many fine-tunes: multi-LoRA

A LoRA fine-tune (Parameter-Efficient Fine-Tuning) adds a pair of small matrices $A$ (input dimension by $r$) and $B$ ($r$ by output dimension) beside each weight, so one base model can serve many fine-tunes at once: each request names its adapter, and a batch mixes requests for different adapters, sharing every pass over the base weights. vLLM does this with --enable-lora (and --max-loras, --max-lora-rank). What it costs, for 64 rank-16 adapters on every projection of the series’ model:

class Multi(torch.nn.Module):
    which = None                                       # adapter per sequence, shared by every layer

    def __init__(self, base):
        super().__init__()
        self.base = base
        self.A = torch.randn(N_ADAPTERS, base.in_features, R, dtype=base.weight.dtype, device=DEVICE) * 0.01
        self.B = torch.randn(N_ADAPTERS, R, base.out_features, dtype=base.weight.dtype, device=DEVICE) * 0.01

    def forward(self, x):                              # x: [batch, tokens, in]
        y = self.base(x)
        if Multi.which is None:
            return y
        if isinstance(Multi.which, str):               # "shared": adapter 0, no gathering
            return y + (x @ self.A[0]) @ self.B[0]
        i = Multi.which
        return y + torch.bmm(torch.bmm(x, self.A[i]), self.B[i])

Full script: lora.py, which wraps every projection of the model and times the three ways. All of this post’s scripts together take about 9 minutes on the same laptop.

one rank-16 adapter on all 7 projections of 28 layers: 35 MiB, against 2.88 GiB for the base model; 1.4% more multiply-adds
decode step, 512 cached tokens per sequence, ms (extra over the base model)
     base    one adapter, shared   gathered, one adapter   gathered, a different adapter each
 16   87.0      90.1 ( +4%)         149.5 (+72%)            150.1 (+72%)
 64  157.0     163.9 ( +4%)         421.4 (+168%)            422.1 (+169%)

An adapter is 35 MiB against the model’s 2.88 GiB, and its products add 1.4% to the projections’ arithmetic; done as two plain multiplies for a shared adapter they added 4% to a step. Mixing adapters the simple way, copying each sequence’s $A_i$ and $B_i$ out of the stack and multiplying in a batch, added 72% at 16 sequences and 169% at 64. The cost is materializing and re-reading one adapter copy per sequence, 0.55 GiB of copies per step at 16 sequences and 2.2 GiB at 64, not the mixing: the same path with one adapter for every sequence costs the same. Punica (Chen et al., 2023) and S-LoRA (Sheng et al., 2023) wrote GPU kernels that read each sequence’s adapter where it lies instead; Punica reports 12 times the throughput of the serving systems it compared against on its multi-adapter workloads, and S-LoRA serves thousands of adapters on one GPU by paging adapter weights in and out alongside the KV cache.

vLLM now

As of version 0.31 (October 2026), vLLM is a PyTorch Foundation project (since May 2025) that runs on NVIDIA, AMD and Intel GPUs and on CPUs, with plugins for Google TPUs, Intel Gaudi and others. Around the loop above it adds structured output (constrained decoding, with backends such as xgrammar and guidance), speculative decoding with draft models, n-grams, Medusa and EAGLE heads (see Speculative Decoding, Derived), quantized models (see Quantization from Scratch), and, since the same scheduler serves them, prefix caching and chunked prefill on by default. On NVIDIA GPUs its attention runs in FlashAttention or FlashInfer kernels that read paged attention’s block tables directly. The defaults that matter most for latency, the token budget and the maximum number of running requests, are chosen per GPU and per entry point, so the same flags behave differently on different machines.

Where this leads

• SGLang and RadixAttention follows SGLang, which organizes cached prefixes as a radix tree and schedules around them.
• TensorRT-LLM follows NVIDIA’s engine, and vLLM vs. SGLang vs. TensorRT-LLM compares the three.

Takeaways

• A forward pass costs about the same for a few sequences as for one, because it reads the weights once; on this laptop, 16 decoding sequences took twice the time of one for 16 times the tokens.
• Static batching wastes that: finished requests hold their slots and new ones wait for a whole batch. On ShareGPT traffic, continuous batching, rescheduling at every step, carried all the traffic up to 1.5 requests a second, two to three times static batches of 128, with median first tokens under half a second; static batches fell behind even at 0.5.
• vLLM V1 schedules prefill and decode by one rule: each request’s computed tokens catch up with its known tokens, running requests first, prompts in chunks under a token budget. A 512-token budget kept the longest gap between tokens at 0.55 s where whole prompts caused gaps of 4.2 s, at the price of slightly later first tokens.
• When the block pool runs out, V1 preempts the newest request and recomputes it later; with an eighth of the memory, throughput fell 9% and some readers waited 29 s, and with a thirty-second, throughput fell 60%.
• At a decode batch of 64, tensor parallelism’s two all-reduces per layer add an eighth to a 70B model’s weight-reading time over NVLink and nearly double it over PCIe, which is why it stays within a machine. Serving many LoRA adapters costs little arithmetic but needs kernels that don’t copy each request’s adapter.

References

Start with the V1 announcement and the scheduler’s source for how the loop works today, then Orca and Sarathi-Serve for why.

• vLLM team, vLLM V1: A Major Upgrade to vLLM’s Core Architecture (2025), and the scheduler, vllm/v1/core/sched/scheduler.py.
• vLLM documentation: Architecture Overview and Optimization and Tuning (preemption, chunked prefill, the token budget).
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI, 2022): iteration-level scheduling.
• Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI, 2024): chunked prefill and generation stalls.
• Anyscale, How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (Anyscale, 2023): the 8 and 23 times throughput comparisons of continuous batching against static batching.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023): the paged KV cache this engine is built on.
• Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019): splitting each layer’s matrices across GPUs, and the two all-reduces per layer.
• Chen et al., Punica: Multi-Tenant LoRA Serving (2023); Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters (2023): kernels that serve many LoRA adapters in one batch.
• NVIDIA, Getting Started with CUDA Graphs: what a CUDA graph is, and the definition quoted above.