Tag: layer-norm

All the articles with the tag "layer-norm".

• The Residual Stream: How a Transformer Turns Tokens Into Predictions

  Updated:19 Oct, 2026

  How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.