Tag: vision-language-models
All the articles with the tag "vision-language-models".
Seeing with Transformers: ViT, CLIP and Vision-Language Models
Updated:Cut an image into patches and it becomes a sequence a Transformer can read; train images against their captions and a picture and its description land near each other in one space. Patches and the patch embedding worked for 224×224 images; the class token, pooling and the inductive bias ViT gives up; CLIP's symmetric contrastive loss and SigLIP's sigmoid loss computed on a 3×3 batch; vision-language models built from an image encoder, a projector and an LLM, and what their image tokens cost in KV cache; and CLIP tested on this blog's own diagrams, where it matches by reading their labels, and a rendering mistake first made it look much better.