Posts
All the articles I've posted.
Beyond the GPU: TPUs, LPUs and What Specialization Buys
Updated:A GPU keeps machinery for any parallel program; specialized chips win by removing it. A systolic array simulated cycle by cycle on a 3×3 matmul, and why it reuses each value read from memory hundreds of times; Google's TPU from the 2017 original to Ironwood, its interconnect and compiler; Groq's LPU, which keeps a model in on-chip SRAM and schedules every cycle at compile time; and the arithmetic of the trade: 609 chips to hold a 70B model in SRAM, against what eight HBM chips can do for one stream of tokens.
Byte-Pair Encoding: How Text Becomes Tokens, One Merge at a Time
Updated:A byte-level BPE tokenizer built from scratch and traced merge by merge: UTF-8 bytes, GPT-2's pre-tokenizer, training by count and encoding by rank, special tokens, the file format, a trainer made fast without changing a merge, 2.2 GB of stories, and what a tokenizer charges Vietnamese.
Chain-of-Thought and Prompt Chaining
Updated:Asking a model for intermediate steps buys serial computation it can't do in one forward pass, and splitting a task into a chain of calls makes those steps inspectable and checkable. Why one forward pass is a fixed-depth circuit, zero-shot and few-shot chain-of-thought, self-consistency, chains with validation between steps, and how errors compound. Then a 1.5B model on problems of 1 to 8 serial steps, answered directly, reasoned in one call, chained one step per call, and chained with each step checked.
Anatomy of a Frontier Model: Reading DeepSeek-V4 Through Part V
Updated:DeepSeek-V4 (April 2026) is two mixture-of-experts models with a million-token context: 1.6 trillion parameters with 49 billion active, and 284 billion with 13 billion active. Read through Part V, each unfamiliar part answers a cost the earlier posts measured: compressed and sparse attention for the KV cache, an indexer for the sparse selection, a widened residual stream kept stable by a doubly stochastic constraint, the Muon optimizer, FP4 experts, and on-policy distillation from more than ten teachers. The KV cache recomputed from the configuration: 2% of a standard model's at a million tokens.