AI Engineering, from the Inside

Where this series starts, where it ends, and how the pieces connect: six words that name the layers of every AI system, a map of the seventeen parts, one running example that grows through every post, the notation used throughout, and a first cost estimate for a single answer.

9 min read

What it actually means for a model to learn: pick a loss, then walk downhill. Worked end to end on real data from this blog, with the loss choice, the closed-form fit, gradient descent and its learning-rate limit, and a first logistic-regression router whose gradient reappears in every later post.

33 min read·Notebook

Give the router 379 features instead of one and it scores 100% on its training data and worse on new questions. Why that happens, why the weights run to infinity, how L2 and L1 penalties stop them (derived, with the soft-threshold that makes L1 exact zeros), why one well-chosen feature matches all 379, and how to tell a real difference from noise when the splits share data.

32 min read·Notebook

An 88.5% accurate router that misses 38% of the questions it exists to catch. Why accuracy hides that, how precision, recall and ranking metrics expose it, how to choose a threshold from what each mistake costs without leaking the test set, whether the router's probabilities mean what they say, and what happens when Vietnamese questions arrive.

27 min read·Notebook

Why a hidden layer can learn what a straight line can't, and how backpropagation gets the gradient of every weight in one backward pass. Worked by hand on XOR with every number, then as matrix rules for a whole layer, checked against finite differences, and costed: where the '6 × parameters FLOPs per training token' rule comes from.

29 min read·Notebook

From two routes to four: the softmax, the loss every language model is trained with and where it comes from in information theory, why its gradient is again just prediction minus target, how to compute it without overflow, label smoothing, a four-way router read through its confusion matrix, and what the same loss costs in memory at a 128,256-token vocabulary.

33 min read·Notebook

Why a deep network's signal shrinks or explodes layer by layer and how He initialization keeps it level, then the optimizers every language model is trained with: momentum and its square-root speed-up, Adam as a per-parameter rescaling with its bias correction derived, why training starts with a warmup, what AdamW's weight decay really changes, and what Adam costs in memory.

39 min read·Notebook

Two layers that sit in almost every network diagram, and what each one actually does: dropout as a regularizer, derived and tested (on its own it makes the router overconfident; alongside weight decay, and in long training, it helps), then normalization, from BatchNorm and why language models can't use it, to LayerNorm and RMSNorm with worked vectors and hand-written backward passes, what normalizing does to the gradient and the learning rate, and what a norm layer costs.

37 min read·Notebook

A deep-learning framework is three things: a tensor library, an autograd tape, and a device model. Strides and views, a 55-line autograd that matches PyTorch to the last bit or two, what the recorded graph looks like, why GPU timings lie without a synchronize, FP32 vs FP16 vs BF16 bit by bit and why BF16 won, the router's training loop matched to the hand-written AdamW, the 16 bytes per parameter measured, and what torch.compile adds.

31 min read·Notebook

A GPU is thousands of simple cores fed by a memory system that is usually the real bottleneck. The H100's layout from its whitepaper, threads, warps and blocks, the roofline and why an operation's FLOPs per byte decides which ceiling it hits, then real kernels written and measured on a laptop GPU: a vector add, coalesced vs scattered reads, and a matrix multiply from naive to tiled, with the arithmetic of why tiles and matrix shapes matter.

28 min read·Notebook

A recurrent network compresses everything it has read into one fixed-size vector, updated one step at a time. Worked by hand and backpropagated through time, then the two problems that sentence contains: gradients that vanish over distance (measured for RNN, GRU and LSTM), and a fixed bottleneck that breaks sequence-to-sequence models (measured on a reversal task, with and without attention). Plus a character-level RNN trained on this blog, the baseline the tiny GPT trained later in the series has to beat.

32 min read·Notebook

An embedding is only as meaningful as the loss that made it, and the data it saw. Word2vec trained on this blog, worked and measured, including the famous analogy test and why it fails here; cosine vs dot product; InfoNCE as cross-entropy over one positive and many negatives, with the temperature and batch size explained; a contrastive head on frozen word2vec vectors, which adds about six and a half points of recall on this little data; and pretrained sentence encoders, one of which matches every English section to its Vietnamese translation.

25 min read·Notebook

A language model is one factorization, p(text) = the product of each token's probability given the ones before it, and training, sampling, caching and evaluation all follow from it. The chain rule worked on this blog, teacher forcing vs generation, what a small model writes when greedy and when sampling, perplexity derived from cross-entropy, why perplexity can't be compared across tokenizers and bits per byte can, measured with a character model and a BPE model on the same text.

20 min read·Notebook

A byte-level BPE tokenizer built from scratch and traced merge by merge: UTF-8 bytes, GPT-2's pre-tokenizer, training by count and encoding by rank, special tokens, the file format, a trainer made fast without changing a merge, 2.2 GB of stories, and what a tokenizer charges Vietnamese.

34 min read·Notebook

Self-attention, masked attention and cross-attention are one computation, softmax(QKᵀ/√d)V, with Q, K and V drawn from different places. Derived from a weighted average, worked through on small matrices, then checked on real models: Qwen2.5-1.5B's mask hides the future exactly, its heads learn to look one token back or at the first token, a translation model's cross-attention lines German words up with English ones, and the cost grows as the square of the length.

18 min read·Notebook

Attention on its own can't tell word order apart, proved in two lines. How position gets in: added sinusoids, learned embeddings, ALiBi's distance penalty, and RoPE's rotation, derived so the query-key score depends only on relative distance. Then a tiny Transformer trained with each, tested on sequences four times longer than it trained on, on one task that needs exact relative position and one that needs to look back over any distance, and what the results say about long-context extension.

21 min read·Notebook

How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.

19 min read·Notebook

Every piece of the series so far, assembled into one 1.8-million-parameter GPT in 80 lines: embeddings, RoPE, RMSNorm, causal attention, a SwiGLU MLP, residual connections. Its parameters counted by formula and by the model, trained on this blog with the settings chosen on a dev slice, scored once on two held-out posts against the LSTM, Kneser–Ney and bz2. Then what each piece is worth, by taking it out, what the GPU is doing, what the model writes, and what it has memorized.

40 min read·Notebook

A language model outputs a probability for every possible next token; decoding is the policy that turns that into text. Greedy and beam search, temperature derived, top-k, top-p and min-p on a real model's distributions, the order the steps run in, repetition penalties, and why the same prompt can give different text. Each one measured with Qwen2.5-1.5B on twenty paragraphs of this blog, against what a person actually wrote next.

34 min read·Notebook

Streaming doesn't make a model generate any faster. It moves the wait the reader feels from the whole answer to the first token. A streaming server on Python's standard library, delivering a real model's tokens three ways: one response, server-sent events, and a WebSocket written by hand. Measured: time to first text, bytes on the wire, reading speed against generation speed, what happens when the user leaves, and how a buffering proxy quietly undoes all of it.

27 min read·Notebook

A model that accepts a long prompt doesn't use all of it equally. The lost-in-the-middle result reproduced on a 1.5B model: a key-value lookup that's perfect at 2,000 tokens and loses things in the middle by 8,000. Why the needle-in-a-haystack test misses it, what RULER and Context Rot measure instead, and the causes: softmax dilution worked out, attention sinks measured, RoPE's reach, and training on short documents. Then what to do about it: put things where they're read, retrieve less, compact, train longer.

22 min read·Notebook

The Transformer of 2017 and a frontier model of 2026 share a skeleton: embeddings, a stack of attention and MLP blocks on a residual stream, an unembedding. What changed is a short list of substitutions, each made because it bought something measurable: pre-norm, RMSNorm, SwiGLU, RoPE, grouped and latent attention, mixtures of experts, and sparse or hybrid attention. Each one, with what it costs and saves, some measured on the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog. Then the scaling laws that decided how big to make them, Kaplan to Chinchilla to the replication that fixed it, and why models are now trained far past compute-optimal.

23 min read·Notebook

A mixture of experts replaces a Transformer's MLP with many MLPs and a router that sends each token to a few of them, so a model can hold far more parameters than it uses per token. The router worked through, the load-balancing loss derived, DeepSeek-V3's bias balancing, capacity and dropped tokens, and shared and fine-grained experts. Then an MoE version of the tiny GPT from Build a Tiny GPT, End to End, and Train It on This Blog, trained four ways, the parameter counts of Mixtral and DeepSeek-V3 recomputed from their configs, and what experts cost at serving time: memory, all-to-all traffic, and batches big enough to keep experts busy.

21 min read·Notebook

Generating text recomputes every earlier token's keys and values unless they are kept. The KV cache keeps them, and it is exact: in Qwen2.5-1.5B-Instruct the next token matched at 32 of 32 steps. It turned a step at 4,096 tokens from 4.7 s into 65 ms on a laptop. Then what it costs: 28 KiB per token for Qwen, 800 KiB for Llama 2 13B, why decoding is limited by memory reads, how batch size trades latency for throughput, and how GQA, MLA and cross-layer sharing shrink the cache.

21 min read·Notebook

Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.

38 min read·Notebook

Standard attention is slow at long contexts not because of its arithmetic but because it writes an n-by-n matrix of scores to memory and reads it back. FlashAttention computes exactly the same result without ever storing that matrix, using tiling and a softmax computed one block at a time. The online softmax derived and checked to within 1e-15, the memory traffic counted, a fused kernel measured against the three-step version, causal masking, and what FlashAttention 2 and 3, and Flash-Decoding, changed.

21 min read·Notebook

DeepSeek-V4 (April 2026) is two mixture-of-experts models with a million-token context: 1.6 trillion parameters with 49 billion active, and 284 billion with 13 billion active. Read through Part V, each unfamiliar part answers a cost the earlier posts measured: compressed and sparse attention for the KV cache, an indexer for the sparse selection, a widened residual stream kept stable by a doubly stochastic constraint, the Muon optimizer, FP4 experts, and on-policy distillation from more than ten teachers. The KV cache recomputed from the configuration: 2% of a standard model's at a million tokens.

17 min read·Notebook

Most production model calls don't need a frontier model, and many don't need text at all: they need a decision. What small means in memory and speed, worked out and checked against a measurement; how small models stay capable (data, overtraining, distillation); and an experiment on this series' own question router, where a 23-million-parameter encoder beats a 1.5-billion-parameter language model on accuracy and calibration, and on speed on an Apple M4 laptop, measured with bootstrap intervals and a careful readout of the language model. Then System One decision models, which return typed answers with probabilities instead of text, with a vendor's claims labeled as such.

18 min read·Notebook

Fine-tuning moves a pretrained model a short distance in weight space, and how far it moves decides what it gains and what it forgets. Continued pretraining vs supervised fine-tuning, the SFT loss with prompt masking worked on a real chat template, learning rates and memory. Then Qwen2.5-0.5B fine-tuned on this blog, two seeds per setting, with what it learns and what it forgets measured side by side; replay as the remedy, EWC derived, and when fine-tuning is the wrong tool.

18 min read·Notebook

A low-rank update is often enough to fine-tune a model, so training only a low-rank part of it cuts trainable parameters and optimizer memory by orders of magnitude, and the result merges back into the weights for free. Intrinsic dimension, LoRA's shapes, initialization and scale, worked memory for an 8B model, QLoRA, prefix and prompt tuning and adapters, and serving many LoRAs at once. Then a hand-written LoRA on the fine-tune from Fine-Tuning, and What It Destroys at ranks 1, 8 and 64, two seeds each: what it learns and what it forgets next to full fine-tuning, how low-rank the full fine-tune's own update is, and a check that merging changes nothing.

25 min read·Notebook

A teacher's full next-token distribution carries far more information per example than the one token the text used. Soft labels and dark knowledge on a real teacher, temperature and the tau-squared factor derived and checked, forward vs reverse KL on a bimodal target, sequence-level and on-policy distillation, and distilled reasoning. Then Qwen2.5-0.5B trained on this blog from hard labels, from a 1.5B teacher's distribution, and from both, two seeds each, with what each learns and forgets.

21 min read·Notebook

When a score isn't differentiable, a model can still climb it with the log-derivative trick. Reinforcement learning's vocabulary mapped onto text, REINFORCE derived in three lines, why a baseline is free and essential, advantages and GAE, and PPO's clipped objective worked case by case. Then REINFORCE, REINFORCE with a baseline and PPO trained on a tiny autoregressive policy, with controls, ten seeds at two learning rates, and the memory bill for PPO on a language model: four models at once.

15 min read·Notebook

Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.

15 min read·Notebook

RLHF's KL-regularized objective has a closed-form optimal policy, and inverting it expresses the reward through the policy itself. Substituted into Bradley-Terry, the partition function cancels and preference learning becomes a logistic loss on log-probability ratios, with no reward model and no RL loop. The derivation step by step, its gradient, an exact check that in the right policy family DPO and RLHF are the same problem, a demonstration of DPO's collapse with a fully flexible policy and deterministic labels, IPO's bounded alternative, and KTO and ORPO.

14 min read·Notebook

When an answer can be checked, the reward model goes away, and sampling a group of answers per question and comparing them within the group gives the advantage without a value network. Verifiable rewards, GRPO's group-relative advantage worked on four samples, its objective and KL estimator, the memory it saves, why groups that all agree teach nothing, and the length bias Dr. GRPO identified, measured on a toy where length never affects the reward.

12 min read·Notebook

A reasoning model spends generated tokens as compute: trained with verifiable rewards, it learns to think before answering, and gets better the longer it thinks. What test-time compute means, how RL produces long chains and distillation passes them to small models, thinking budgets, and what 1,000, 10,000 and 50,000 thinking tokens cost in KV cache and time on two real models. Then a 1.5B model on checkable word problems, two seeds: answering directly, reasoning step by step, and voting over up to 16 chains, with the tokens each one costs, then on harder eight-step problems, where voting stops helping and a verifier could still gain.

18 min read·Notebook

Asking a model for intermediate steps buys serial computation it can't do in one forward pass, and splitting a task into a chain of calls makes those steps inspectable and checkable. Why one forward pass is a fixed-depth circuit, zero-shot and few-shot chain-of-thought, self-consistency, chains with validation between steps, and how errors compound. Then a 1.5B model on problems of 1 to 8 serial steps, answered directly, reasoned in one call, chained one step per call, and chained with each step checked.

13 min read·Notebook

API prompt caching is the KV cache and prefix sharing sold as a price. What is actually cached, why only an exact prefix can hit, breakpoints and lifetimes, the write and read prices and the break-even worked from published multipliers, the time a cached prefix saves measured on a 1.5B model, the gigabytes a provider holds for one, how to order a prompt, and how agents break their own cache.

13 min read·Notebook

An agent's window fills with tool output, not with its own thinking. Three API primitives handle that differently: clearing replaces old tool results with stubs, compaction replaces the transcript with a summary, and memory moves notes outside the window. Each one measured on Claude Sonnet 5.5, with what it costs and what it loses.

25 min read·Notebook

Finding the nearest embeddings is easy and linear; every index trades a little recall for a lot of speed. Similarity metrics and when they agree, why trees fail in high dimensions, and LSH, IVF, product quantization with a worked 8-byte code, and HNSW as a navigable small world, with an interactive graph search. Then all of them measured with FAISS on 71,865 real sentence embeddings: recall against latency and bytes per vector.

16 min read·Notebook

Keyword search matches the words a query uses; dense retrieval matches what it means. BM25 derived and worked on this blog's sections, dense retrieval and its truncation, reciprocal rank fusion, and the metrics. Then a benchmark built from the blog itself, English and Vietnamese: fusion helps in English and hurts in Vietnamese, diacritics matter, and technical Vietnamese can be searched with English keywords, with bootstrap intervals.

15 min read·Notebook

Retrieval has to be cheap per document because it scores every document; reranking can be expensive per document because it only sees the top few. The bi-encoder and cross-encoder, their cost per query computed, ColBERT's late interaction and MaxSim worked token by token, why max and not mean, hard negatives, index sizes with ColBERTv2's compression, and LLM rerankers. Then cross-encoder reranking measured on the English and Vietnamese benchmark of Semantic and Hybrid Search, with what it gains and what it costs.

15 min read·Notebook

Most RAG failures happen before generation: the chunk that holds the answer is never retrieved. Eight ways to cut this blog into chunks, plus a control, measured on 150 questions by whether the answer's sentence comes back; HyDE, measured; an embedding cache keyed by model and text; a semantic cache's hits and wrong answers at six thresholds; and when RAG is the wrong tool.

21 min read·Notebook

An agent is a model in a loop, with tools and a rule for stopping. What function calling is (a protocol around generated text: the model never runs anything), how a call is emitted and parsed, parallel calls, think-act-observe, stopping, and the transcript as append-only context, with its cost growing quadratically per turn unless cached. Then an agent searching this blog, on 1.5B and 3B models, measured against one retrieval and one answer: whether small models choose to use tools, what forcing a call fixes, and how they cite what they found.

17 min read·Notebook

Three ways to spend a model's tokens on thinking around tool use: interleave reasoning and actions (ReAct), plan everything first and then execute (plan-and-execute), or answer, critique and retry (reflection). How each works, the failure each is prone to, and all three run on the same 20 tool-use questions over a fictional database with a 3B model, measuring accuracy, tokens and time.

16 min read·Notebook

An agent's context is its working memory; everything else is storage with a retrieval policy. A taxonomy of agent memory, the four operations and the rule each needs, Generative Agents' recency, importance and relevance score worked, MemGPT's tiers, forgetting, memory as files, and privacy. Then retrieval policies measured on 250 conversations whose facts change, with ablations: how often each shows the model an outdated fact, and what fixing that costs.

11 min read·Notebook

When one retrieval pass can't answer the question, there are three ways to go further: let the model decide what to fetch next (agentic RAG), fetch structure instead of chunks (GraphRAG), or navigate a document's own outline instead of embedding it (vectorless retrieval). How each works and what it costs, with GraphRAG's indexing bill computed for this blog, and keyword, dense, hybrid and LLM-navigated retrieval compared on 30 questions about it.

16 min read·Notebook

A recursive language model never reads its long input: the input sits in a Python REPL as a variable, and the model writes code to inspect it and calls itself on the pieces. How the idea works, why its three design choices matter, what the paper measured, and a miniature built and run on this blog's whole archive with a 3B model, against direct reading and a fixed map-over-chunks program.

20 min read·Notebook

Three answers to one question, how an agent gets capabilities without bloating its context: the Model Context Protocol for live tools and data, Skills for procedures loaded on demand, and the Open Knowledge Format for curated knowledge. A working MCP server for this blog, called through the SDK and as raw JSON-RPC under the stateless 2026-07-28 protocol; progressive disclosure's token arithmetic; an OKF bundle generated from the posts; and the security model all three share.

18 min read·Notebook

Splitting a task across agents buys clean contexts and parallelism, and costs coordination and lost information. Why subagents exist; roles; communication topologies; the A2A protocol; the standard orchestration patterns; a token-cost model, on real document sizes, of one agent against a lead with six workers (a context a quarter to two-fifths the size and far fewer sequential steps whatever the assumptions, a token bill that can go either way, and only 8% of the source reaching the lead); and the published evidence on when multi-agent systems win and when they lose.

13 min read·Notebook

What if the orchestrator of a multi-agent system were itself a model, trained to pick and instruct frontier models? Sakana's Fugu report (2026) answers with two designs: a router that reads one hidden state and picks one worker, trained by supervised fine-tuning and then evolution strategies, and a Conductor trained with GRPO to write whole multi-agent workflows in natural language. How each is built, the failure called orchestration collapse and its fix, and the reported results read critically: ahead of the best single model on 7 of 11 benchmarks each, against baselines gathered from providers and leaderboards rather than run under the same conditions.

14 min read·Notebook

The model sets the ceiling; the harness around it decides how close an agent gets. The parts of a harness; loop design (stopping conditions, retries, budgets); graphs and when they beat a loop; and the definition of done as a check the harness runs instead of a claim the model makes. Measured on a small agent: a verifier that checks only where the answer came from, against the same agent without it, with success at each step budget; and what Anthropic's long-running harnesses do about agents that declare victory early.

14 min read·Notebook

Agent frameworks are an opinion about state and control flow. LangChain's runnables composed with a pipe turn out to be function composition with a shared interface; LangGraph's state graph is a loop with typed state, reducers and a checkpoint after every step. The same agent written by hand and with LangGraph (LangGraph 1.2, langchain-core 1.6): the same 10 lines with LangGraph's prebuilt pieces, under half a millisecond of overhead per run, and what that buys: pausing for a person's approval, editing a pending tool call, resuming, and replaying from any saved step.

11 min read·Notebook

Coding agents are the most demanding agent harnesses in production: large codebases, many tools, real side effects, and a definition of done that can actually be executed. How Claude Code and Cursor work, using only what their documentation and engineering posts state: the gather-act-verify loop, searching a codebase with grep and with an embedding index (and why Cursor dropped the index), verification, project memory, context management, permissions and checkpoints, subagents and hooks (with a working hook), and Cursor's tab and apply models.

14 min read·Notebook

Time to first token is a prefill problem and time per output token is a decode problem; serving both on one GPU makes them interfere, and splitting them trades that interference for a KV-cache transfer and a ratio to get right. The metrics and SLOs, goodput, chunked prefill, and disaggregation, then a step-by-step simulation of an 8B model on H100s under Poisson traffic, measuring every gap between tokens: why averages hide stalls, why chunked prefill sized to the target wins in an idealized model, and what disaggregation offers that the model leaves out.

13 min read·Notebook

After sharing key-value heads, two levers are left for the KV cache: store fewer tokens, or store each in fewer bits. Both are lossy in different ways. The budget worked for Llama-2-13B, eviction from sliding windows to attention sinks, heavy hitters and SnapKV, and per-token versus per-channel quantization as in KIVI; then both measured on a real model's loss on held-out text, against the full cache.

13 min read·Notebook

A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.

25 min read·Notebook

A serving engine is a loop that decides, every forward pass, which requests' tokens go into it. What a pass costs on a real GPU, measured; static batching against continuous batching; vLLM V1's scheduler, where prefill and decode are one rule under a token budget, written out and run on ShareGPT traffic with the measured costs, with paged memory and preemption; the processes vLLM runs; tensor parallelism's communication, worked for a 70B model; and serving many LoRA fine-tunes from one model, measured.

23 min read·Notebook

Decoding is memory-bound, so checking k guessed tokens costs about as much as generating one. A rejection-sampling rule makes the output exactly the target model's distribution whatever the guesser, proved and checked by sampling. The expected tokens per step derived and checked, the speedup with the draft's cost, and an interactive simulator; draft models, prompt lookup, Medusa heads and EAGLE; and acceptance measured on real models for two kinds of task.

18 min read·Notebook

Quantization maps floats onto a small integer grid with a scale and a zero-point; everything else is about where the rounding error lands. A worked 4-bit example, per-tensor vs per-channel vs group-wise scales, and GPTQ and AWQ implemented from their papers and run on two real layers of Qwen2.5-0.5B with their real inputs: what each buys at 8, 4 and 3 bits, why GPTQ needs enough calibration data, and why AWQ helps only where activations have outliers.

18 min read·Notebook

Local inference is the quantization and memory-bandwidth story at consumer scale: GGUF packages a quantized model in one memory-mappable file, and llama.cpp turns bytes per second into tokens per second. Inside a real GGUF file, what K-quant names like Q4_K_M mean, and tokens per second measured on this laptop's GPU and CPU against the bandwidth prediction, with what each quantization level costs in quality.

12 min read·Notebook

Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.

16 min read·Notebook

TensorRT-LLM began by compiling each model into an engine: operations fused into fewer kernels, and every kernel timed and chosen for the GPU and the shapes it would serve. Both steps measured on a laptop GPU: fusion removed 26% of the operations per token and 7% of prefill time; the best version of one kernel depended on the shape, and the worst cost twice as much. Then why NVIDIA moved the project onto a PyTorch runtime, and what it keeps of the compiled design.

17 min read·Notebook

vLLM, SGLang and TensorRT-LLM now share one design: continuous batching over a paged KV cache with prefix reuse, run from PyTorch with CUDA graphs. What differs is defaults, hardware and ecosystem. Measured: on multi-turn chat with room for the working set, the default prefix caches reused 92.0% of prompt tokens in SGLang, 90.9% in vLLM and 92.0% in TensorRT-LLM; with a smaller cache, TensorRT-LLM's fell furthest. The rest is each engine's source, read at its latest release.

14 min read·Notebook

An eval is a measurement instrument, and it needs what any instrument needs: validity, reliability and a known error bar. What to measure and with which metric; benchmarks and contamination; how many questions it takes to tell two models apart (8,077 for 2 points, or as few as 1,570 when the same questions are scored for both); confidence intervals and Cohen's kappa worked; an LLM judge measured on questions with known answers, where a 1.5B judge is sound on questions it can answer, falls back on position on ones it can't, and is fooled by claims of having checked; and arena ratings as Bradley-Terry with their intervals.

17 min read·Notebook

An agent's answer can be right for the wrong reasons, wrong for a trivial one, or right only some of the time. Outcome, trajectory and tool-use evaluation; SWE-bench and τ-bench; pass@k against pass^k, with the arithmetic and unbiased estimators; and a small agent instrumented with OpenTelemetry's GenAI span conventions and evaluated from its traces, where most 'failures' turned out to be the answer's format or the harness's own token budget, and the traces showed exactly which.

15 min read·Notebook

A language model reads instructions and data through the same channel, so text in a retrieved page can act as a command. Input and output guardrails and classifier filters; direct versus indirect injection; why there is no escaping, as there is for SQL; and an injection attack measured against a small tool-using assistant that summarizes this blog, before and after four defenses, from a system-prompt rule and spotlighting to an input classifier and removing the tool.

17 min read·Notebook

Sampling leaves slack: many tokens would do at most steps. A keyed bias toward a pseudo-random green list spends that slack on a signature anyone with the key can test for, with no access to the model. The green-list scheme and its z-test derived and worked, how many tokens detection needs, the scheme run on a real 1.5B model at four bias strengths with its cost in likelihood, why low-entropy text resists it, what substitution and paraphrase do to it, distortion-free schemes (Aaronson's Gumbel trick, SynthID-Text's tournament), and why post-hoc AI-text detectors are a different and weaker thing.

21 min read·Notebook

Cut an image into patches and it becomes a sequence a Transformer can read; train images against their captions and a picture and its description land near each other in one space. Patches and the patch embedding worked for 224×224 images; the class token, pooling and the inductive bias ViT gives up; CLIP's symmetric contrastive loss and SigLIP's sigmoid loss computed on a 3×3 batch; vision-language models built from an image encoder, a projector and an LLM, and what their image tokens cost in KV cache; and CLIP tested on this blog's own diagrams, where it matches by reading their labels, and a rendering mistake first made it look much better.

16 min read·Notebook

A computer-use agent operates software the way a person does: it looks at the screen, decides, and clicks or types. The perceive-think-act loop; screenshots versus accessibility trees, measured on real pages (a screenshot is 1,000 to 4,800 tokens; a raw accessibility tree 4,000 to 170,000, pruned 600 to 20,000); coordinate grounding; the action space; what a task costs as screenshots pile up; OSWorld and how far agents have come; and why safety, not reasoning, decides where these agents can be deployed.

13 min read·Notebook

A language model samples by writing the distribution down one token at a time. For images that means hundreds of thousands of sequential steps in an arbitrary pixel order, so two other answers grew up: learn an encoder and maximize a bound on the likelihood (the VAE), or learn a critic and fool it (the GAN). The ELBO derived and checked exactly on a model small enough to solve, the reparameterization trick, the KL term worked by hand, the GAN's optimal discriminator and its Jensen-Shannon value checked numerically, and all three models (VAE, GAN, WGAN-GP) trained on the same eight-mode test, where off-mode samples, a dropped mode and an outright training failure all show up, with a test of where the VAE's off-mode samples come from.

27 min read·Notebook

Corrupt data a little at a time until it is pure noise, and train a network to undo one step. For images the corruption is Gaussian noise; for text it is masking. The forward process and its closed form, the noise schedule worked numerically, the ε-prediction loss derived from the ELBO of VAEs and GANs, DDPM and DDIM sampling, classifier-free guidance, and latent diffusion; a diffusion model trained on the eight-mode test of VAEs and GANs, with step counts and guidance measured; then masked diffusion for text, where a toy language shows exactly what decoding many tokens in parallel gets wrong, and the FLOPs and latency arithmetic against autoregressive decoding.

29 min read·Notebook

A GPU keeps machinery for any parallel program; specialized chips win by removing it. A systolic array simulated cycle by cycle on a 3×3 matmul, and why it reuses each value read from memory hundreds of times; Google's TPU from the 2017 original to Ironwood, its interconnect and compiler; Groq's LPU, which keeps a model in on-chip SRAM and schedules every cycle at compile time; and the arithmetic of the trade: 609 chips to hold a 70B model in SRAM, against what eight HBM chips can do for one stream of tokens.

15 min read·Notebook

Where a model runs is an optimization over cost, latency, privacy and quality, and routing turns it from one decision per product into one per query. The arithmetic of cloud against device; what fits in a phone's memory; hybrid designs like Apple's; routers, cascades and learned routing; how routing relates to mixture-of-experts; and a cascade measured between a 1.5B and a 3B model on 150 checkable questions, where routing by the kind of question beat routing on the small model's confidence.

13 min read·Notebook

A voice agent has about one second between the user stopping and the agent starting, and five components share it. Requirements and a worked latency budget stage by stage, where end-of-turn detection turns out to be the biggest item; transport over WebRTC; streaming speech recognition and synthesis; the LLM's time to first token; cascaded versus speech-to-speech models; barge-in and tool calls mid-turn; and capacity for 1,000 simultaneous calls from a simulation of continuous batching, which needs far fewer GPUs than intuition says because speech is slow.

18 min read·Notebook

Three research directions for what comes after next-token prediction, each with what is shown and what is hoped. World models learn a simulator and train or plan inside it: a learned pendulum imagined forward drifts 3° in 15 steps and 15° in 100, one reason imagination horizons are short. JEPA predicts embeddings instead of pixels: on a toy world, the naive version collapses to a point, and a stop-gradient, an EMA target with a predictor, or VICReg's variance terms prevent it. And self-improvement: what systems like AlphaEvolve have actually done, separated from what remains speculation.

16 min read·Notebook

The series' running example, built as one working system: a question-answering assistant for this blog, with a semantic cache, hybrid retrieval, a cross-encoder reranker, a prompt that fences retrieved text as data, a 3B model that cites its passages, and a citation check. One question traced through every stage; a scorecard on 150 questions, 15 the blog doesn't answer, 30 repeats and a planted injection, with the judge checked by hand; the cost per answer, refining the series map's estimate; and what to change next.

19 min read·Notebook