AI Engineering, from the Inside: The Map

This series goes from a logistic regression to a language-model system that is served, evaluated and guarded. It is written for engineers who can code and want to understand these systems from the inside. It assumes Python, dot products and matrix multiplication, derivatives, and probability at the level of “a distribution sums to one”. Everything else, including tools like eigenvalues, is explained where it’s first needed.

Each post explains one idea from the problem that made it necessary: how it works step by step, the math with a worked example, what it costs with the arithmetic shown, runnable code, and where it fails. Each concept has one home. Every other post that needs it links back with a one-sentence recap rather than teaching it again. The full list of posts, by part, is on the series page.

The six words are layers of one system, built roughly from the model outward, and the rest of the series is the machinery around them. Many later decisions exist to change one number: what an answer costs to serve.

At a glance

Six words name six layers of one system: LLM, quantization, fine-tuning, RAG, agent and MCP, spread over seventeen parts in six stages.
One answer from an 8B model costs about 1.5 GPU-seconds served alone: 32 ms to read a 2,000-token prompt and 1.43 s to write 300 tokens, almost all of it spent moving 16 GB of weights.
Batching 64 readers makes it 18 times cheaper, about 0.08 GPU-seconds per answer, because every decode step reads the weights once for all 64.
Memory caps the batch at about 212 conversations on an 80 GB H100, since each 2,300-token conversation’s cache takes 0.30 GB.

Open Table of contents

Six words, six layers
The map
The running example: the Notebook Assistant
What one answer costs
Notation
Where this leads
Takeaways
References

Six words, six layers

Six words come up in every conversation about AI systems. They aren’t six separate topics. They are layers of one system, roughly from the model outward:

Word	What it is	Where it lives in this series
LLM	A model that predicts the next token, trained to minimize a loss	Parts I–VI build one from a logistic regression upward
Quantization	Storing the model in fewer bits so it’s cheaper to run	Part XII
Fine-tuning	Continuing that training on new data or preferences	Part VII
RAG	Retrieving relevant text and putting it in the prompt	Part IX
Agent	The model in a loop, calling tools until a task is done	Parts X–XI
MCP	A protocol for giving an agent tools and context	Part X

The rest of the series is the machinery around those layers:

how text is generated (Part IV);
how models are served (Part XII);
how they’re evaluated (Part XIII) and attacked (Part XIV);
what else they can see (Part XV);
the hardware and system design underneath them (Part XVI).

The map

The seventeen parts fall into six stages. Each builds on the ones before it, and the arrows show the main dependencies. Serving a model (Part XII) depends directly on how the model is built, not only on what’s built with it.

Six stages of the series in two rows. Top: Foundations (Parts I–II) leads to The Model (III–VI), which leads to Adapting It (VII–VIII). Adapting It leads down to Building With It (IX–XI), which leads left to Running It (XII–XIV); The Model also leads directly down to Running It. Running It leads left to Beyond (XV–XVII).

Within each stage, the order of the parts matters less than the arrows. Every post links back to the specific earlier ideas it uses, so a reader can start at any stage and follow the links back to what it depends on.

The running example: the Notebook Assistant

Every idea in the series is applied to one system: an assistant that answers readers’ questions about this blog, in English and Vietnamese. It’s a good test case because its data is real and ours to use, it needs every layer in the table above, and being bilingual makes it a harder engineering problem. Tokenization, retrieval and evaluation all behave differently across the two languages.

It grows one step per post:

Part	The Notebook Assistant gains…
I	a router that sends each question to the right part of the blog, first as a logistic regression, then as a small neural network
II	embeddings of every post section, and a first language model trained on the blog
III–V	a tiny GPT trained on the blog’s own text, then modernized with the design choices of current models
VI–VII	a real small open model, fine-tuned with LoRA on questions and answers from the posts, then aligned with preference data
VIII–IX	a system prompt and retrieval over the blog, with hybrid English/Vietnamese search and a reranker
X–XI	tools and memory: it can search the posts, run a post’s code, and expose the blog through an MCP server
XII	a real server with a measured memory budget, plus a quantized build that runs on a laptop
XIII–XIV	a bilingual evaluation set, an LLM judge checked against human labels, and defenses against a deliberately poisoned post
XV–XVI	the ability to answer questions about the blog’s diagrams, and a decision about where it should run

What one answer costs

It helps to know from the start roughly what the finished assistant costs to run, because so many later decisions exist to change this number. Take one question answered by an 8-billion-parameter model shaped like Llama-3-8B (32 layers, 8 key/value heads of dimension 128), stored in BF16 at 2 bytes per parameter, on one H100 SXM GPU. That GPU does 989 TFLOPS of dense BF16 math and reads memory at 3.35 TB/s. Retrieval puts 2,000 tokens of posts in the prompt, and the answer is 300 tokens long.

Reading the prompt (prefill) costs about $2 \times$ parameters FLOPs per token: $2 \times 8 \times 10^9 \times 2{,}000 = 3.2 \times 10^{13}$ FLOPs. At 989 TFLOPS that is 32 ms.
Writing the answer (decode) produces one token per step, and each step has to read all $8 \times 10^9 \times 2$ bytes $= 16$ GB of weights from memory. At 3.35 TB/s that’s 4.78 ms a step, so 300 tokens take 1.43 s.

That’s about 1.5 GPU-seconds for one reader’s answer, and almost all of it is spent moving weights, not doing math.

Now serve 64 readers at once. Each decode step still reads the weights once, now shared by all 64 answers. It also reads every reader’s own cache of past tokens. That cache is $32 \text{ layers} \times 8 \text{ heads} \times 128 \text{ dims} \times 2\ (K \text{ and } V) \times 2 \text{ bytes} = 128$ KiB per token, or 0.30 GB for a 2,300-token conversation.

One step: $16 + 64 \times 0.30 \approx 35$ GB of reads, which takes 10.5 ms.
Per answer: 300 steps produce 64 answers, so decode costs $300 \times 10.5 / 64 \approx 49$ ms per answer. Add each answer’s own 32 ms of prefill, which batching doesn’t share, and it comes to about 0.08 GPU-seconds, 18 times cheaper.

At an assumed $2.50 per H100-hour (rental prices vary widely), that’s about $0.001 per answer served alone and $0.00006 at a batch of 64: roughly 1,000 versus 18,000 answers per dollar.

Two caveats keep these numbers honest.

They are ideal lower bounds. Real kernels reach only a fraction of the GPU’s peak compute and bandwidth, and Part XII: Inference engineering measures how much.
Memory caps the batch. With 80 GB on the card and 16 GB of weights, only about $(80 - 16) / 0.30 \approx 212$ conversations’ caches fit at once, so the batch can’t grow without limit.

Every number in this section is derived later in the series.

The FLOP count comes from how a Transformer is built (The Residual Stream counts a block’s FLOPs), and Backpropagation, by Hand derives the $2N$ forward and $6N$ training FLOPs per token, which is why training costs three times as much per token as this forward pass.
Decode’s memory bound, and the cache that caps the batch size, are the subject of The KV Cache.
What changes each of these numbers: smarter memory management, reusing shared prompt prefixes, speculative decoding and quantization, all in Part XII: Inference engineering.

Notation

The same symbols mean the same things in every post:

Symbol	Meaning
$x$ , $X$	an input vector; a batch or sequence stacked as rows, so a layer computes $Y = XW$
$W$ , $b$	a weight matrix and a bias
$\theta$	all trainable parameters together
$z$	logits, the scores before a sigmoid or softmax
$p_\theta(\cdot)$	the model’s probability distribution
$\mathcal{L}$	a loss
$\eta$	the learning rate
$\tau$	a temperature, one symbol wherever temperature appears
$B$ , $S$ , $T$	batch size; tokens already in context; tokens processed in this pass
$D$ , $F$	model width; the MLP’s hidden width
$L$	number of layers
$N$ , $K$ , $H$	query heads; key/value heads; head dimension, with group size $G = N/K$
$V$	vocabulary size

Later parts add a few more ( $E$ and $k$ for mixture-of-experts, $r$ for LoRA rank, $\beta$ and $\pi$ for alignment), each introduced where it’s first used. Symbols that belong to a single post, like $n$ and $d$ for a dataset’s size and dimension or a Hessian and its eigenvalues, are defined where they appear.

A few terms are used strictly.

A token is a unit of the tokenizer, not a word.
Logits are always the scores before the sigmoid or softmax.
Throughput is tokens per second across a whole batch.
Latency is time per generated token (TPOT, time per output token).
TTFT, time to first token, is how long before the first token appears.

Where this leads

The series reads best in order for a reader starting from scratch. Every post links back to what it depends on, so it also works to start anywhere and follow the links back. Some posts that already existed on this blog before the series, like The Transformer’s Three Attention Patterns, The KV Cache and Byte-Pair Encoding, are part of it. They are the home for their concepts, and the new posts link to them rather than repeating them.

Learning Is Optimization starts with the most basic question: what does it mean for a model to learn?

Takeaways

The six words (LLM, quantization, fine-tuning, RAG, agent, MCP) are layers of one system, not six topics, and the seventeen parts follow them from the model outward.
One running example, the Notebook Assistant, grows by one step per post, from a logistic-regression router to a served, evaluated, guarded system.
Decoding an answer is bound by memory, not math: almost all of an answer’s 1.5 GPU-seconds goes to reading weights, which is why batching makes it 18 times cheaper.
The same symbols mean the same things in every post, and terms like token, logits, throughput and latency are used strictly.

References

Start with the H100 datasheet, which gives the two hardware numbers every cost estimate in this post rests on.

NVIDIA, H100 Tensor Core GPU (datasheet): the hardware numbers, 989 TFLOPS dense BF16 and 3.35 TB/s of HBM3 bandwidth for the SXM variant; the PCIe card is lower.
Grattafiori et al. (Meta AI), The Llama 3 Herd of Models (arXiv, 2024): the 8B model’s shape, 32 layers, 8 key/value heads, head dimension 128.