Anatomy of a Frontier Model: Reading DeepSeek-V4 Through Part V

Part V: Modern architecture took a Transformer apart by cost. How LLM Architecture Evolved, 2017–2026 listed the substitutions; Mixture of Experts opened the mixture of experts; The KV Cache sized the KV cache; Attention Alternatives surveyed the alternatives to full attention; FlashAttention showed how attention is computed without materializing its scores. This post reads one frontier model through all of it: DeepSeek-V4, whose technical report appeared in April 2026 (DeepSeek-AI, 2026). (In September DeepSeek released V4.1-Flash, with a different design; everything here is V4.0, as its report and released weights define it.)

The report describes two models, both mixtures of experts with a context of one million tokens: DeepSeek-V4-Pro, 1.6 trillion parameters with 49 billion active per token, and DeepSeek-V4-Flash, 284 billion with 13 billion active. It calls them a preview. Most of the design is familiar from earlier DeepSeek models; what’s new is aimed at one target, the million-token context, and the report’s headline is efficiency there: at a million tokens, V4-Pro needs 27% of DeepSeek-V3.2’s per-token compute and 10% of its KV cache.

Table of contents

The skeleton, and what’s kept

Start with what the list from How LLM Architecture Evolved, 2017–2026 predicts, and it holds. V4 is a decoder with a residual stream, an RMSNorm before each sublayer, SwiGLU experts, rotary positions and a 128K-token vocabulary, trained to predict the next token (plus, as in V3, a second token through one extra multi-token-prediction layer).

Two rows already say a lot. A single key-value head shared by every query head is multi-query attention taken to the limit (see How LLM Architecture Evolved, 2017–2026), and each cached entry serves as both the key and the value. And 384 experts with 6 chosen per token is the fine-grained MoE of Mixture of Experts: per token, Pro uses about 3% of its parameters.

The attention: compress, then select

At a million tokens, the dilution from Long Context, and Where It Breaks and the cache from The KV Cache both say the same thing: an entry for every earlier token, in every layer, can’t be kept and attended to. V4 alternates two kinds of attention layer, both reading compressed entries rather than tokens.

• Compressed Sparse Attention (CSA). Every 4 tokens become one cache entry: a learned, softmax-weighted pooling of their key-value vectors (computed per channel, with learned position biases; each entry pools 8 tokens, its own block and the one before, so consecutive entries overlap, which the report doesn’t explain). So the cache grows at a quarter of the token rate. Then, instead of attending to all those entries, each query attends to only the most relevant 1,024 of them (512 in Flash), chosen by a small lightning indexer: a cheap scoring network, 64 heads of 128 dimensions, run in FP4 after post-training’s quantization-aware training, that ranks the compressed blocks for each query. This is DeepSeek-V3.2’s sparse attention applied on top of compression.
• Heavily Compressed Attention (HCA). Every 128 tokens become one entry, without overlap, and queries attend to all of them densely: at a million tokens that’s about 8,000 entries, presumably few enough that selecting among them isn’t worth it.

From the released configuration (the report says only “interleaved”), Pro has 30 CSA and 31 HCA layers, starting with two HCA; Flash starts with two plain sliding-window layers, then 21 CSA and 20 HCA. Three details make it work:

• A sliding window in every layer. A query can’t see the uncompressed tokens of its own block, since a compressed entry is formed only once the block is complete. So every CSA and HCA layer also attends to the last 128 tokens directly. Short-range detail comes from the window, long-range context from the compressed entries: the hybrid of Attention Alternatives, inside each layer.
• An attention sink. Each head has a learned logit whose exponential is added to its softmax’s denominator, so a head’s attention can sum to less than one, even nearly zero (Long Context, and Where It Breaks’s sinks, made explicit, as in Attention Alternatives’s learned sinks; the report cites OpenAI, 2025, and Xiao et al.). In the online softmax of FlashAttention it is just one more term in the running denominator.
• QK-norm and partial RoPE. Queries and compressed entries are RMS-normalized per head, the stability trick How LLM Architecture Evolved, 2017–2026 found becoming standard, and RoPE rotates only the last 64 of each 512-dimensional vector: a cousin of MLA’s separate 64-dimensional rotated key (Where Am I? Position in Attention, from Sinusoids to RoPE), but inside the entry rather than beside it, leaving 448 position-free dimensions that can be stored in FP8. Because the cached entry is also the value, it carries absolute position into the output, so V4 rotates each output back by its own position, leaving only relative position. (The released configuration extends RoPE with YaRN (see Where Am I? Position in Attention, from Sinusoids to RoPE), which the report doesn’t mention.)

The KV cache, recomputed

# DeepSeek-V4's KV cache per token, from its published configuration (the
# report, arXiv 2606.19348, plus the released config.json and FlashMLA's
# cache layout), against DeepSeek-V3.2 and a plain GQA baseline.
#
# Per layer, V4 keeps one compressed key/value entry (it is both: a single
# shared KV head) for every m tokens:
#   CSA layers: m = 4, plus an indexer key per entry for the sparse selection;
#   HCA layers: m = 128, no indexer.
# An entry is 512 numbers: 448 in FP8 plus the 64 RoPE dimensions in BF16,
# with 8 bytes of scales (7 one-byte UE8M0 scales + 1 pad): 584 bytes.
# The indexer key is 128 numbers in FP4 (64 bytes) plus scales, taken here
# as 4 bytes (one 1-byte scale per 32).
# Every layer also attends to a sliding window of the last 128 tokens; that
# cache is fixed in size, so per token of a 1M context it is negligible.
ENTRY, INDEXER, WINDOW, CONTEXT = 584, 64 + 4, 128, 2**20

def v4(csa_layers, hca_layers, total_layers):
    per_token = csa_layers * (ENTRY + INDEXER) / 4 + hca_layers * ENTRY / 128
    window = (total_layers + 1) * WINDOW * ENTRY / CONTEXT              # +1: the multi-token-prediction layer
    return per_token + window

pro, flash = v4(30, 31, 61), v4(21, 20, 43)
# V3.2: MLA on all 61 layers, one 656-byte entry per token per layer
# (512 FP8 + 16 bytes of scales + 64 BF16 RoPE dims), plus an FP8 indexer key
# of 128 numbers and a 4-byte scale.
v32 = 61 * (656 + 132)
# GQA baseline: 8 key-value heads of 128 dimensions, keys and values, BF16.
gqa = lambda layers: layers * 2 * 8 * 128 * 2

The rest of the script prints each model’s bytes per token, its cache at a million tokens, the ratios between them, and Pro’s split between CSA and HCA layers.

Full script: kv_cache.py

DeepSeek-V4-Pro      5,036 bytes per token,   4.9 GiB at 1M tokens; 10.5% of V3.2's, 2.0% of a 61-layer GQA-8 model's
DeepSeek-V4-Flash    3,517 bytes per token,   3.4 GiB at 1M tokens;  7.3% of V3.2's, 2.0% of a 43-layer GQA-8 model's
DeepSeek-V3.2       48,068 bytes per token,  46.9 GiB at 1M tokens
GQA-8, 61 layers   249,856 bytes per token, 244.0 GiB at 1M tokens
V4-Pro: 97% of its cache is in the 30 CSA layers; the 31 HCA layers hold 3%

The recomputed cache matches the report: about 5 KB per token for Pro, 10.5% of V3.2’s (the report says 10%), and 2% of a conventional model with 8 key-value heads (the report: “approximately 2%”). At a million tokens Pro’s cache is about 5 GB per sequence, where the same depth with GQA would need 244 GiB (262 GB), over three H100s’ worth of memory for one sequence. Of that factor of 50, about 7 is per entry (one shared key-value head, mostly in FP8, 584 bytes where GQA stores 4,096 per layer and token) and about 7 along the sequence.

The arithmetic shows where the cache that remains lives. Almost all of it, 97%, is in the CSA layers: a 584-byte entry plus the small indexer key every 4 tokens. The HCA layers, at one entry per 128 tokens, cost almost nothing. And against V3.2, which already used a compressed latent (MLA, How LLM Architecture Evolved, 2017–2026), the gain is compression along the sequence, on top of compression along the width.

The experts

The experts are V3’s design, from Mixture of Experts, with adjustments:

• More experts, fewer chosen. Pro routes each token to 6 of 384 experts of width 3,072 (V3: 8 of 256, width 2,048), plus one shared expert every token uses. All 61 layers are MoE: the first three route by a hash of the token instead of a learned router, where V3’s first three were dense.
• A different router score. The affinity is $\sqrt{\text{softplus}(x)}$ rather than V3’s sigmoid.
• Bias balancing, unchanged from V3: a bias update speed of 0.001 plus V3’s tiny per-sequence balance loss (weight 0.0001), which stops any one sequence from piling onto a few experts.
• Hash routing in the first three layers. Instead of dense MLPs, the first layers are MoE layers whose experts are chosen by a fixed function of the token’s identity (Roller et al., 2021). The report gives no reason; a plausible one is that this early, a token’s representation is still mostly its own identity, so a learned router has little to add.
• No limit on target nodes, which V3 had imposed to bound the all-to-all traffic counted in Mixture of Experts. The report also describes a fused communication-and-compute kernel for expert parallelism, 1.5 to 1.7 times faster in inference.

A wider residual stream: mHC

The Residual Stream is one vector per token that every sublayer reads from and adds to. Hyper-connections (Zhu et al., 2025) widen it: keep $n$ copies of the stream, here $n = 4$, and let learned matrices mix them, $X_{l+1} = B_l X_l + C_l F_l(A_l X_l)$, where $A_l$ reads one input for the sublayer from the 4 copies, $C_l$ writes its output back into them, and $B_l$ mixes the copies with each other. The sublayers themselves are unchanged: they still see a $d$-dimensional vector. It’s a new axis to scale along that costs little compute.

Plain hyper-connections train unstably when stacked deep, and The Residual Stream already said why: the identity path is what keeps a deep stack trainable, and an unconstrained $B_l$ can amplify or cancel the signal at every layer. mHC (Xie et al., 2026) constrains $B_l$ to be doubly stochastic, non-negative with every row and every column summing to 1, by running 20 steps of Sinkhorn–Knopp normalization (alternately rescaling rows and columns). A doubly stochastic matrix has spectral norm at most 1, so it can’t amplify the stream, and a product of them is still doubly stochastic, so the guarantee holds through 61 layers: it’s a mixture, never a gain. $A_l$ and $C_l$ go through a sigmoid to stay positive and bounded.

The cost is memory and communication rather than arithmetic: four copies of the stream are four times the residual stream’s activations. The report puts mHC’s wall-time overhead at 6.7% of an overlapped pipeline stage, after fused kernels, selective recomputation and a reworked pipeline schedule. It gives no ablation showing what mHC buys; the mHC paper is where to look for that.

The optimizer: Muon

The AdamW of Initialization and Optimizers scales each parameter’s update by its own running statistics. Muon (Jordan et al., 2024) treats each weight matrix as a whole: it takes the momentum of the gradient matrix and replaces it with the nearest semi-orthogonal matrix, the $UV^\top$ of its singular value decomposition, so the update has every singular value equal to 1 and moves the matrix equally in every direction it moves at all. Computing the SVD each step is too slow, so a few iterations of a Newton–Schulz polynomial approximate it with matrix multiplies. V4 uses 10: eight with coefficients chosen to converge fast, then two that pin the singular values to 1. Following Liu et al. (2025), the update is rescaled to a fixed RMS of 0.18 so AdamW’s learning rates and weight decay carry over.

Muon updates the large matrices; AdamW still handles the embeddings, the output head, the norms’ gains and mHC’s small parameters. One trick from other Muon users isn’t needed: QK-Clip, which Kimi K2 used to cap attention logits that grow too large, is redundant here because QK-norm already keeps them bounded. How LLM Architecture Evolved, 2017–2026’s “what hasn’t changed” noted Muon starting to replace AdamW in 2025; V4 is a frontier model trained with it, like Kimi K2 before it. As with mHC, the report claims faster convergence and better stability without ablations.

Precision: FP8 everywhere, FP4 experts

• FP8 for training and for weights, as in V3: weights ship in FP8 with a scale per 128×128 block.
• FP4 for the experts and the indexer. The routed experts’ weights, the bulk of the 1.6T parameters, are stored in 4-bit floats, as is the indexer’s query-key path. This is learned, not just applied afterwards: FP4 quantization-aware training during post-training, where the forward pass sees the 4-bit weights and gradients pass straight through the rounding (the precision of PyTorch from the Inside, and Quantization from Scratch to come). Turning FP4 back into FP8 for the multiply is exact, because FP8’s format has two more exponent bits than FP4’s (as long as the scales of the small FP4 blocks inside each FP8 block stay within range), so on today’s hardware the multiply runs at FP8 speed while the weights take half the memory.
• Mixed precision in the cache: the 64 RoPE dimensions in BF16, the rest in FP8, which is the 584-byte entry above.

Training, and after

Pre-training, 32 to 33 trillion tokens, grows the context in stages: 4K, 16K, 64K, then 1M tokens. Flash trains with dense attention for its first trillion tokens, then warms up the indexer and switches to sparse attention at 64K. The report doesn’t give the compute used.

Stability is a section of its own. Loss spikes were tied to outliers in the MoE layers, which the routing made worse. Two fixes: clamping SwiGLU’s linear branch to $[-10, 10]$ and capping its gate at 10, and anticipatory routing, computing a step’s expert choices with slightly older parameters, switched on automatically when a spike is detected. The report is candid that “their underlying principles remain insufficiently understood”.

Post-training replaces V3.2’s mixed reinforcement-learning stage with on-policy distillation. Specialist models are trained per domain (math, code, agents, instruction following) with supervised fine-tuning and then reinforcement learning (GRPO, from GRPO and Reinforcement Learning from Verifiable Rewards). Then a single student learns from more than ten of these teachers at once: it generates its own answers, and is trained to match each teacher’s full next-token distribution on them, by reverse KL over the whole vocabulary. Knowledge Distillation explains why the student sampling its own text, and the reverse direction of the KL, matter.

Reasoning modes, Non-think, High and Max, aren’t a token budget setting. They are trained in: specialists with different length penalties and context windows during reinforcement learning, plus, for Max, a system prompt asking for maximum effort. Reasoning Models and Test-Time Compute covers what spending more tokens on reasoning buys.

What the report doesn’t settle

A technical report is not a set of ablations, and this one says less than it seems to in several places:

• No ablations for the headline components. CSA and HCA against V3.2’s attention, mHC against a plain residual stream, Muon against AdamW: the benefits are claimed, not measured in the report.
• The 27% compute figure can’t be reproduced from the report: it’s “estimated”, in “equivalent FP8 FLOPs”, without the formula.
• The cache accounting depends on details the report omits. The 10% and 2% hold under any reasonable byte layout (10 to 13%, 2.0 to 2.2%), but the near-exact match to the report’s 9.5 and 13.7 times needs the indexer key stored in FP4 and V3.2’s indexer in FP8, which come from the released code and kernels rather than the text; the reference code itself keeps the indexer cache in BF16.
• It’s a preview, and already superseded. DeepSeek released V4.1-Flash in September 2026 (a new causal encoder–decoder design and a new cache format), and its API now routes V4-Pro requests to it.

None of this makes the design less interesting. It makes the report a description of choices, to be weighed with the evidence for each from the papers behind them.

Where this leads

• Small and Specialized Models opens Part VI: Kinds of models, turning from how models are built to which kind to use: small and specialized models, and models that don’t generate text at all.
• Knowledge Distillation explains the ideas behind V4’s on-policy distillation; Quantization from Scratch comes back to FP4 and quantization.

Takeaways

• DeepSeek-V4 keeps the 2023 skeleton: pre-norm RMSNorm, SwiGLU experts, RoPE, a mixture of experts with shared and fine-grained experts and bias balancing. Pro uses 49B of its 1.6T parameters per token.
• Its attention compresses the KV cache along the sequence: every 4 tokens into one entry in CSA layers, which then attend sparsely to 1,024 indexer-selected entries, and every 128 into one in HCA layers, attended densely, each with a 128-token sliding window and a learned sink.
• Recomputed from the configuration, its cache is about 5 KB per token, 10% of V3.2’s and 2% of an 8-head GQA model’s: about 5 GB per sequence at a million tokens instead of about 260 GB.
• mHC widens the residual stream to 4 copies, kept stable by doubly stochastic mixing, which can’t amplify; Muon replaces AdamW for the large matrices; experts are stored in FP4, learned with quantization-aware training.
• Post-training distills more than ten specialist teachers into one student on its own samples, by reverse KL.
• The report claims rather than ablates its main components: read it as a set of design choices, each backed by its own paper.

References

Start with the DeepSeek-V4 report and its released configuration, then the V3 technical report it builds on and the mHC paper for the widened residual stream.

• DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (2026): the report, for the design and its claims; configurations from the released config.json.
• DeepSeek-AI, DeepSeek-V3 Technical Report (2024): the base it builds on, for the experts and bias balancing.
• Xie et al., mHC: Manifold-Constrained Hyper-Connections (2026): the doubly stochastic constraint on the widened residual stream.
• Zhu et al., Hyper-Connections (ICLR, 2025): widening the residual stream.
• Jordan et al., Muon: An optimizer for hidden layers in neural networks (2024): the optimizer.
• Liu et al., Muon is Scalable for LLM Training (2025): weight decay and update-RMS matching at scale.
• Roller, Sukhbaatar, Szlam and Weston, Hash Layers For Large Sparse Models (NeurIPS, 2021): routing by a hash of the token.
• Xiao, Tian, Chen, Han and Lewis, Efficient Streaming Language Models with Attention Sinks (ICLR, 2024): attention sinks.
• OpenAI, gpt-oss-120b & gpt-oss-20b Model Card (2025): the “OpenAI, 2025” the V4 report cites for learned attention sinks.
• Kimi Team, Kimi K2: Open Agentic Intelligence (2025): MuonClip and its QK-Clip, the trick V4 doesn’t need.