Tag: infonce
All the articles with the tag "infonce".
Embeddings and Contrastive Learning: Making Similar Mean Close
Updated:An embedding is only as meaningful as the loss that made it, and the data it saw. Word2vec trained on this blog, worked and measured, including the famous analogy test and why it fails here; cosine vs dot product; InfoNCE as cross-entropy over one positive and many negatives, with the temperature and batch size explained; a contrastive head on frozen word2vec vectors, which adds about six and a half points of recall on this little data; and pretrained sentence encoders, one of which matches every English section to its Vietnamese translation.