What actually happens inside a transformer, layer by layer.
The residual stream is the shared path every layer reads from and writes to. This notebook traces it — attention, KV caches, quantization, and the inference engines built to serve it fast. Mechanism first, benchmarks second. Written as I go, in English and Vietnamese.
Featured
Attention Alternatives: Sparse, Linear, and Recurrent
Updated:Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.
Context Engineering: Compaction, Tool Clearing, and Memory
Updated:An agent's window fills with tool output, not with its own thinking. Three API primitives handle that differently: clearing replaces old tool results with stubs, compaction replaces the transcript with a summary, and memory moves notes outside the window. Each one measured on Claude Sonnet 5.5, with what it costs and what it loses.
Predicting Chess Outcomes: A Disciplined ML Walkthrough
Updated:Predicting who wins a Lichess blitz game from its opening, rebuilt end to end on 100,000 games. Against a rating curve refitted to the data, the opening adds nothing at move 3 and a Brier gain of about 0.002 at move 10, on all seven splits tried; a Stockfish evaluation makes it 0.0084. The method is the point: an audited dataset, a leakage contract enforced by tests, a baseline worth beating, and diagnostics before features.
The KV Cache: How Transformers Avoid Recomputing the Past
Updated:Generating text recomputes every earlier token's keys and values unless they are kept. The KV cache keeps them, and it is exact: in Qwen2.5-1.5B-Instruct the next token matched at 32 of 32 steps. It turned a step at 4,096 tokens from 4.7 s into 65 ms on a laptop. Then what it costs: 28 KiB per token for Qwen, 800 KiB for Llama 2 13B, why decoding is limited by memory reads, how batch size trades latency for throughput, and how GQA, MLA and cross-layer sharing shrink the cache.
Paged Attention: Managing the KV Cache Like Virtual Memory
Updated:A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.
The Residual Stream: How a Transformer Turns Tokens Into Predictions
Updated:How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.
SGLang and RadixAttention: Sharing the KV Cache Across Requests
Updated:Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.
The Transformer's Three Attention Patterns: Self, Masked, and Cross
Updated:Self-attention, masked attention and cross-attention are one computation, softmax(QKᵀ/√d)V, with Q, K and V drawn from different places. Derived from a weighted average, worked through on small matrices, then checked on real models: Qwen2.5-1.5B's mask hides the future exactly, its heads learn to look one token back or at the first token, a translation model's cross-attention lines German words up with English ones, and the cost grows as the square of the length.
vLLM Architecture: Continuous Batching and Serving at Scale
Updated:A serving engine is a loop that decides, every forward pass, which requests' tokens go into it. What a pass costs on a real GPU, measured; static batching against continuous batching; vLLM V1's scheduler, where prefill and decode are one rule under a token budget, written out and run on ShareGPT traffic with the measured costs, with paged memory and preemption; the processes vLLM runs; tensor parallelism's communication, worked for a 70B model; and serving many LoRA fine-tunes from one model, measured.
vLLM vs. SGLang vs. TensorRT-LLM: Comparing LLM Serving Architectures
Updated:vLLM, SGLang and TensorRT-LLM now share one design: continuous batching over a paged KV cache with prefix reuse, run from PyTorch with CUDA graphs. What differs is defaults, hardware and ecosystem. Measured: on multi-turn chat with room for the working set, the default prefix caches reused 92.0% of prompt tokens in SGLang, 90.9% in vLLM and 92.0% in TensorRT-LLM; with a smaller cache, TensorRT-LLM's fell furthest. The rest is each engine's source, read at its latest release.
AI Engineering, from the Inside: The Map
Updated:Where this series starts, where it ends, and how the pieces connect: six words that name the layers of every AI system, a map of the seventeen parts, one running example that grows through every post, the notation used throughout, and a first cost estimate for a single answer.
Byte-Pair Encoding: How Text Becomes Tokens, One Merge at a Time
Updated:A byte-level BPE tokenizer built from scratch and traced merge by merge: UTF-8 bytes, GPT-2's pre-tokenizer, training by count and encoding by rank, special tokens, the file format, a trainer made fast without changing a merge, 2.2 GB of stories, and what a tokenizer charges Vietnamese.
TensorRT-LLM: Compiling Inference Engines Ahead of Time
Updated:TensorRT-LLM began by compiling each model into an engine: operations fused into fewer kernels, and every kernel timed and chosen for the GPU and the shapes it would serve. Both steps measured on a laptop GPU: fusion removed 26% of the operations per token and 7% of prefill time; the best version of one kernel depended on the shape, and the worst cost twice as much. Then why NVIDIA moved the project onto a PyTorch runtime, and what it keeps of the compiled design.
Recent Posts
Features and Overfitting: How a Model Memorizes, and How L1 and L2 Stop It
Updated:Give the router 379 features instead of one and it scores 100% on its training data and worse on new questions. Why that happens, why the weights run to infinity, how L2 and L1 penalties stop them (derived, with the soft-threshold that makes L1 exact zeros), why one well-chosen feature matches all 379, and how to tell a real difference from noise when the splits share data.
How a GPU Runs Deep Learning: Memory, the Roofline, and a First Kernel
Updated:A GPU is thousands of simple cores fed by a memory system that is usually the real bottleneck. The H100's layout from its whitepaper, threads, warps and blocks, the roofline and why an operation's FLOPs per byte decides which ceiling it hits, then real kernels written and measured on a laptop GPU: a vector add, coalesced vs scattered reads, and a matrix multiply from naive to tiled, with the arithmetic of why tiles and matrix shapes matter.
Initialization and Optimizers: Where to Start, and How to Step
Updated:Why a deep network's signal shrinks or explodes layer by layer and how He initialization keeps it level, then the optimizers every language model is trained with: momentum and its square-root speed-up, Adam as a per-parameter rescaling with its bias correction derived, why training starts with a warmup, what AdamW's weight decay really changes, and what Adam costs in memory.
PyTorch from the Inside: Tensors, Autograd and Precision
Updated:A deep-learning framework is three things: a tensor library, an autograd tape, and a device model. Strides and views, a 55-line autograd that matches PyTorch to the last bit or two, what the recorded graph looks like, why GPU timings lie without a synchronize, FP32 vs FP16 vs BF16 bit by bit and why BF16 won, the router's training loop matched to the hand-written AdamW, the 16 bytes per parameter measured, and what torch.compile adds.