Tag: scheduling

All the articles with the tag "scheduling".

• vLLM Architecture: Continuous Batching and Serving at Scale

  Updated:19 Oct, 2026

  A serving engine is a loop that decides, every forward pass, which requests' tokens go into it. What a pass costs on a real GPU, measured; static batching against continuous batching; vLLM V1's scheduler, where prefill and decode are one rule under a token budget, written out and run on ShareGPT traffic with the measured costs, with paged memory and preemption; the processes vLLM runs; tensor parallelism's communication, worked for a 70B model; and serving many LoRA fine-tunes from one model, measured.