Tag: residual-stream
All the articles with the tag "residual-stream".
The Residual Stream: How a Transformer Turns Tokens Into Predictions
Updated:How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.