Seeing with Transformers: ViT, CLIP and Vision-Language Models

Everything so far in this series has been text, but nothing in the Transformer (The Transformer’s Three Attention Patterns) is specific to text. It takes a sequence of vectors and returns one. Turn an image into a sequence of vectors and the same architecture can see. This post covers that step, the training that ties images to language, and how today’s models that read pictures are put together.

Table of contents

Modalities

A modality is a kind of input: text, images, audio, video. Each needs a way to become vectors, which is what an encoder does for it, and the interesting question for multimodal systems is how vectors from different modalities are made comparable. There are two answers in this post: train two encoders so their outputs share one space (CLIP), or translate one modality’s vectors into the other’s token space and let a language model read them (vision-language models).

Patches: an image as a sequence

The Vision Transformer (ViT; Dosovitskiy et al., 2020) uses the most direct translation. Cut the image into a grid of small square patches, flatten each patch’s pixels into one vector, and project every patch with the same learned linear layer, the patch embedding. Add position embeddings (Where Am I? Position in Attention, from Sinusoids to RoPE) so the model knows where each patch came from (ViT learns one embedding per position in the flattened sequence; later models use 2-D-aware schemes such as 2-D RoPE so the same model can take images of different sizes), and the result is a sequence like any other:

# An image as a sequence. ViT-B/16 on a 224x224 RGB image: cut it into
# 16x16 patches, flatten each (16 * 16 * 3 numbers), and project each to the
# model's width with one shared linear layer, the patch embedding.
H = W = 224; P = 16; C = 3; D = 768
n = (H // P) * (W // P)
print(f"{H}x{W} image, {P}x{P} patches: {H // P} x {W // P} = {n} patches, each {P * P * C} numbers")
print(f"patch embedding: a {P * P * C} x {D} matrix + bias = {P * P * C * D + D:,} parameters")
print(f"sequence for the Transformer: {n} patch tokens + 1 [CLS] = {n + 1} positions, attention over {(n + 1) ** 2:,} pairs")
for res, p in ((224, 14), (336, 14), (448, 14), (1024, 16)):
    print(f"  {res}x{res} with {p}x{p} patches: {(res // p) ** 2:,} tokens")

# In a vision-language model the patch embeddings, after a projector, become
# tokens in the language model's context, and cost KV cache like text.
# LLaVA-1.5: a CLIP ViT-L/14 at 336x336 gives 576 image tokens, fed to a 7B
# language model (32 layers, width 4096, full multi-head attention, 16-bit).
kv_per_token = 32 * 2 * 4096 * 2                 # layers x (K and V) x width x bytes
print(f"\nLLaVA-1.5 (7B language model): {kv_per_token / 2 ** 10:.0f} KiB of KV cache per token")
print(f"  one image, 576 tokens: {576 * kv_per_token / 2 ** 20:.0f} MiB; a 20-image conversation: "
      f"{20 * 576 * kv_per_token / 2 ** 30:.1f} GiB")
print(f"  as text, 576 tokens is about {int(576 * 0.75)} English words")

Full script: patches.py

224x224 image, 16x16 patches: 14 x 14 = 196 patches, each 768 numbers
patch embedding: a 768 x 768 matrix + bias = 590,592 parameters
sequence for the Transformer: 196 patch tokens + 1 [CLS] = 197 positions, attention over 38,809 pairs
  224x224 with 14x14 patches: 256 tokens
  336x336 with 14x14 patches: 576 tokens
  448x448 with 14x14 patches: 1,024 tokens
  1024x1024 with 16x16 patches: 4,096 tokens

LLaVA-1.5 (7B language model): 512 KiB of KV cache per token
  one image, 576 tokens: 288 MiB; a 20-image conversation: 5.6 GiB
  as text, 576 tokens is about 432 English words

A 224×224 image in 16×16-pixel patches is 196 tokens (each “word” in the paper’s title, “an image is worth 16×16 words”, is one such patch), plus one extra. That extra is the [CLS] token, borrowed from BERT: a learned vector prepended to the sequence whose output, after every layer of attention to the patches, serves as the representation of the whole image. The alternatives are to average the outputs of all patch tokens (mean pooling), which the ViT paper found works as well once tuned, or to pool with a small attention layer (SigLIP does this). Models that feed a language model keep all the patch outputs instead.

Inductive bias. A convolutional network builds in two assumptions: that nearby pixels matter most to each other (locality), and that a pattern means the same thing wherever it appears (translation equivariance). ViT builds in almost none: from the first layer, every patch can attend to every other, and spatial structure has to be learned from data. The paper found that this costs ViT on small datasets, where a CNN’s assumptions are worth a lot, and pays off on very large ones (it was pretrained, with labels, on about 300 million images), where learned structure beats assumed structure. Part of that gap was the training recipe: DeiT (Touvron et al., 2021) later trained ViTs well on ImageNet’s 1.3 million images with heavy augmentation and distillation. The same trade, data for priors, is the story of Transformers in text.

ViTs also learn well without labels or captions. MAE (He et al., 2021) masks most patches and trains the model to reconstruct them; DINO (Caron et al., 2021) and DINOv2 (Oquab et al., 2023) train a ViT to match its own outputs on different views of an image, the self-distillation of World Models, JEPA and Recursive Self-Improvement’s JEPA family. DINOv2’s features capture spatial detail that CLIP’s miss, and some vision-language models use both encoders side by side.

CLIP: images and text in one space

A classifier learns a fixed set of labels. CLIP (Radford et al., 2021) learns from 400 million image–caption pairs collected from the web, with no labels at all, using the contrastive objective of Embeddings and Contrastive Learning. An image encoder (a ViT) and a text encoder (a Transformer) each produce a vector, both normalized to unit length. In a batch of $N$ pairs, the $N \times N$ matrix of cosine similarities should be large on the diagonal (each image with its own caption) and small elsewhere. The loss treats each row as a classification problem (which of these $N$ captions is mine?) and each column likewise (which image is mine?), with cross-entropy over a softmax, and averages the two directions: the symmetric InfoNCE loss. Worked on three pairs:

import numpy as np

# CLIP's loss on a batch of 3 image-caption pairs. Embeddings are unit
# vectors; S[i, j] is the cosine similarity of image i and caption j, and the
# logits are S divided by a temperature. The matching pairs are on the diagonal.
S = np.array([[0.90, 0.20, 0.10],
              [0.30, 0.80, 0.40],
              [0.05, 0.35, 0.70]])
t = 0.07                                   # CLIP's initial temperature
L = S / t
def log_softmax(x, axis):
    x = x - x.max(axis=axis, keepdims=True)
    return x - np.log(np.exp(x).sum(axis=axis, keepdims=True))
img_to_txt = -np.diag(log_softmax(L, axis=1))     # each image: which caption is mine? (cross-entropy over the row)
txt_to_img = -np.diag(log_softmax(L, axis=0))     # each caption: which image is mine? (over the column)
print("logits = similarities / 0.07:\n", L.round(2))
print("image -> caption losses:", img_to_txt.round(4), " caption -> image losses:", txt_to_img.round(4))
print(f"CLIP loss (symmetric InfoNCE) = mean of both directions = {(img_to_txt.mean() + txt_to_img.mean()) / 2:.4f}")

# SigLIP: no softmax over the batch. Every pair is its own binary question,
# match (label +1) or not (-1), scored with a sigmoid; a learned scale t
# (from log t, initialized at log 10) and bias b (initialized at -10).
tp, b = 10.0, -10.0
z = np.where(np.eye(3) == 1, 1, -1)
pair = np.log1p(np.exp(-z * (tp * S + b)))        # -log sigmoid(z * logit)
print("\nSigLIP per-pair losses:\n", pair.round(4))
print(f"SigLIP loss = sum over all 9 pairs / 3 images = {pair.sum() / 3:.4f}")

Full script: clip_loss.py

(The cosines here are illustrative; a trained CLIP’s similarities between matching pairs are typically far smaller, around 0.2 to 0.35, and it learns the temperature, as the logarithm of its inverse, capped so the scale never exceeds 100.)

logits = similarities / 0.07:
 [[12.86  2.86  1.43]
 [ 4.29 11.43  5.71]
 [ 0.71  5.   10.  ]]
image -> caption losses: [0.0001 0.0041 0.0068]  caption -> image losses: [0.0002 0.0018 0.0139]
CLIP loss (symmetric InfoNCE) = mean of both directions = 0.0045

SigLIP per-pair losses:
 [[1.3133e+00 3.0000e-04 1.0000e-04]
 [9.0000e-04 2.1269e+00 2.5000e-03]
 [1.0000e-04 1.5000e-03 3.0486e+00]]
SigLIP loss = sum over all 9 pairs / 3 images = 2.1647

Dividing the similarities by a temperature of 0.07 (CLIP’s starting value; it is learned) turns differences of a few tenths into logit gaps of 5 to 10, so this well-aligned batch has a loss near zero. The softmax couples every pair in the batch: each caption competes with all the others. That is what makes large batches valuable (more negatives per example) and what makes them expensive, since the softmax needs the whole similarity matrix, which gets complicated to compute when the batch is split across many devices.

SigLIP (Zhai et al., 2023) removes the coupling. Each of the $N^2$ pairs becomes its own yes-or-no question, scored with a sigmoid: is this image this caption’s? Matching pairs are labeled +1 and all others −1, with a learned scale (multiplying the similarity, the inverse of a temperature) and a learned bias. No softmax means no normalization over the batch, so every pair can be scored independently, and it trains better than CLIP’s loss at batch sizes below about 16,000 (both saturate around 32,000). The worked numbers show a detail of its design: at the paper’s starting values (scale 10, from a learned log-scale initialized at $\log 10$, and bias −10), even a matching pair with similarity 0.9 gets a negative logit and a loss of 1.3. That is deliberate. In a batch of $N$, only $N$ of the $N^2$ pairs match; starting near the prior of “no match” keeps the huge initial loss from all those negatives from driving large, destabilizing first steps.

Once trained, CLIP does things no classifier could. Zero-shot classification: embed the image and the captions “a photo of a dog”, “a photo of a cat”, and so on, and pick the closest. Cross-modal search: find images by text and text by image. And its image encoder became the eyes of most of the models below.

CLIP on this blog’s diagrams

CLIP learned from photographs and their captions. This blog’s diagrams are schematic: boxes, arrows and labels. Can it still match each one to the alt text its post gives it? Each diagram is rendered whole into a 512×512 square, once as drawn and once with every text label hidden. render.py does this with headless Chrome, called at its macOS install path, so as written it runs only on a Mac; the folder ships its renders as clip-diagram-renders.tar.gz instead.

Full script: render.py

clip_diagrams.py takes each diagram’s caption from the alt text in the blog’s posts as of that run, embeds images and captions with CLIP ViT-B/32, and counts how often the right match is ranked first, in both directions:

with torch.no_grad():
    def image(folder):               # each tower's pooled output, then CLIP's projection into the shared space
        px = proc(images=[Image.open(f"{folder}/{n}.png").convert("RGB") for n in names], return_tensors="pt")
        return model.visual_projection(model.vision_model(pixel_values=px["pixel_values"]).pooler_output)
    def text(captions):              # CLIP reads at most 77 tokens; longer alt texts are cut
        enc = proc(text=captions, return_tensors="pt", padding=True, truncation=True)
        out = model.text_model(input_ids=enc["input_ids"], attention_mask=enc["attention_mask"])
        return model.text_projection(out.pooler_output)
    caps = {"the post's alt text": [alts[n] for n in names],
            "the file name as words": [n.replace("-", " ") for n in names]}
    print(f"{len(names)} diagrams; chance of the right match at random: 1 in {len(names)} ({1 / len(names):.0%})\n")
    print(f"{'diagrams':20s} {'caption':24s} {'caption -> diagram':>19s} {'diagram -> caption':>19s}  (right one first)")
    for folder, label in (("fit", "as drawn"), ("notext", "with labels hidden")):
        img = image(folder)
        for name, c in caps.items():
            S = torch.nn.functional.normalize(img, dim=-1) @ torch.nn.functional.normalize(text(c), dim=-1).T
            i2t = int((S.argmax(1) == torch.arange(len(names))).sum())
            t2i = int((S.argmax(0) == torch.arange(len(names))).sum())
            print(f"{label:20s} {name:24s} {t2i:>12d} of {len(names)} {i2t:>12d} of {len(names)}")

Full script: clip_diagrams.py

19 diagrams; chance of the right match at random: 1 in 19 (5%)

diagrams             caption                   caption -> diagram  diagram -> caption  (right one first)
as drawn             the post's alt text                 5 of 19            4 of 19
as drawn             the file name as words              4 of 19            3 of 19
with labels hidden   the post's alt text                 1 of 19            2 of 19
with labels hidden   the file name as words              1 of 19            2 of 19

alt texts: median 40 tokens, 3 of 19 cut at 77

On an Apple M4 laptop, on the CPU, the CLIP comparison takes about 2 minutes.

Given an alt text, CLIP ranks the right diagram first for 5 of the 19, and the right alt text first for 4: better than the one in 19 chance would give, not by much, and with 19 items the uncertainty is wide. Hide the labels and it falls to chance. Whatever CLIP recognizes in these diagrams, it recognizes in their text: CLIP is known to read words in images (so well that a printed label can override what a photo shows, the “typographic attack” of Goh et al., 2021), and boxes and arrows carry little else it was trained on.

The first version of this experiment scored much higher, from thumbnails made with macOS’s Quick Look. Those thumbnails turned out to be crops: the top-left corner of each diagram at full size, its labels large and legible, the rest cut off. The model was reading enlarged text, which also explains why the score fell with faithful renders. Looking at the inputs a model actually receives is part of evaluating it. A vision-language model that reads text in images properly should do much better on documents like these than one CLIP vector, though that wasn’t measured here.

Vision-language models

A vision-language model (VLM) lets a language model read images alongside text. The common recipe, popularized by LLaVA (Liu et al., 2023), has three parts:

1. a pretrained image encoder, usually a CLIP or SigLIP ViT, which turns the image into a grid of patch vectors (keeping all of them, not just one pooled vector);
2. a small projector, a linear layer or two-layer MLP, that maps each patch vector into the language model’s embedding space;
3. a pretrained language model, which receives the projected patches as if they were tokens, placed in the prompt where the image goes.

Training is staged: first only the projector, on image–caption pairs, with the language model frozen, so the projector learns to map image features into the language model’s embedding space; then the projector and the language model together on instruction data (questions and answers about images). The image encoder stays frozen in both. Larger systems train end to end, handle high resolutions by tiling an image into crops or by giving the encoder native-resolution input with 2-D position encodings, and sometimes skip a separate encoder altogether. A different family (Flamingo, Llama 3.2 Vision) doesn’t put image tokens in the context at all: the language model attends to image features through added cross-attention layers, which changes the cost picture below.

What image tokens cost

Those tokens cost what text tokens cost. In the output above, LLaVA-1.5’s CLIP ViT-L/14 at 336×336 produces 576 image tokens, and its 7B language model stores 512 KiB of KV cache per token (The KV Cache): 288 MiB for one image, about what 430 words of text would take, and 5.6 GiB for a conversation with twenty images. Higher resolution multiplies it: 1,024 tokens at 448×448, 4,096 at 1024×1024 with 16-pixel patches. This is why VLMs compress image tokens (pooling neighboring patches, or learned resamplers that turn hundreds of patches into a few dozen query tokens), why APIs cap image resolution, and why a computer-use agent’s screenshots dominate its context (Computer-Use Agents).

Where this leads

• Computer-Use Agents feeds a model screenshots, where these image tokens dominate the context.
• World Models, JEPA and Recursive Self-Improvement trains a Vision Transformer to predict the embeddings of hidden image blocks (I-JEPA).
• The Notebook Assistant, End to End answers only from the blog’s text, and names this post’s image embeddings as what answers about its diagrams would need.

Takeaways

• ViT turns an image into a sequence: 16×16 patches of a 224×224 image are 196 tokens, each a linear projection of 768 pixel values, plus a [CLS] token or mean pooling for the whole image.
• ViT gives up a CNN’s locality and translation priors and wins them back with data, the same trade Transformers made in text.
• CLIP trains an image and a text encoder with a symmetric contrastive loss over the batch; SigLIP scores every pair independently with a sigmoid, removing the batch-wide softmax.
• On this blog’s diagrams, CLIP matched alt texts to the right diagram 5 times in 19 and fell to chance with the labels hidden: it reads their text. A cropped rendering had first made it look much better.
• A vision-language model is an image encoder, a projector and a language model; its image tokens cost KV cache like text, 288 MiB per image for LLaVA-1.5.

References

Start with the ViT paper and CLIP, then SigLIP for the sigmoid loss and LLaVA for the encoder-projector-LLM recipe most open VLMs follow.

• Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020): the Vision Transformer, its patch embedding and its data-for-priors trade.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021): the symmetric contrastive loss and zero-shot classification.
• Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP, 2023): the per-pair sigmoid loss.
• Touvron et al., Training data-efficient image transformers & distillation through attention (DeiT, 2021): training ViTs on ImageNet alone.
• He et al., Masked Autoencoders Are Scalable Vision Learners (MAE, 2021); Caron et al., Emerging Properties in Self-Supervised Vision Transformers (DINO, 2021); Oquab et al., DINOv2 (2023): learning ViT features without labels or captions.
• Goh et al., Multimodal Neurons in Artificial Neural Networks (2021): CLIP reading text in images, and the typographic attack.
• Liu et al., Visual Instruction Tuning (LLaVA, 2023); Liu et al., Improved Baselines with Visual Instruction Tuning (LLaVA-1.5, 2023): the encoder, projector and LLM recipe, its staged training, and the 576 image tokens.