Tag: multimodal
All the articles with the tag "multimodal".
Computer-Use Agents: Screens as Observations, the Mouse as a Tool
Updated:A computer-use agent operates software the way a person does: it looks at the screen, decides, and clicks or types. The perceive-think-act loop; screenshots versus accessibility trees, measured on real pages (a screenshot is 1,000 to 4,800 tokens; a raw accessibility tree 4,000 to 170,000, pruned 600 to 20,000); coordinate grounding; the action space; what a task costs as screenshots pile up; OSWorld and how far agents have come; and why safety, not reasoning, decides where these agents can be deployed.
Seeing with Transformers: ViT, CLIP and Vision-Language Models
Updated:Cut an image into patches and it becomes a sequence a Transformer can read; train images against their captions and a picture and its description land near each other in one space. Patches and the patch embedding worked for 224×224 images; the class token, pooling and the inductive bias ViT gives up; CLIP's symmetric contrastive loss and SigLIP's sigmoid loss computed on a 3×3 batch; vision-language models built from an image encoder, a projector and an LLM, and what their image tokens cost in KV cache; and CLIP tested on this blog's own diagrams, where it matches by reading their labels, and a rendering mistake first made it look much better.