Tag: embeddings
All the articles with the tag "embeddings".
Embeddings and Contrastive Learning: Making Similar Mean Close
Updated:An embedding is only as meaningful as the loss that made it, and the data it saw. Word2vec trained on this blog, worked and measured, including the famous analogy test and why it fails here; cosine vs dot product; InfoNCE as cross-entropy over one positive and many negatives, with the temperature and batch size explained; a contrastive head on frozen word2vec vectors, which adds about six and a half points of recall on this little data; and pretrained sentence encoders, one of which matches every English section to its Vietnamese translation.
The Residual Stream: How a Transformer Turns Tokens Into Predictions
Updated:How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.