Computer-Use Agents: Screens as Observations, the Mouse as a Tool

Most agents in this series act through APIs: a tool is a function with a schema (Giving Agents Tools and Knowledge). Most software has no such API. It has a screen meant for people. A computer-use agent works the way a person does: it takes a screenshot, decides what to do, and moves the mouse, clicks, types or presses keys, then looks again. Any application becomes reachable, at the price of a much harder perception problem and a much larger attack surface.

Table of contents

The loop

Perceive, think, act, the agent loop of From Function Calling to the Agent Loop with a screen in place of tool results:

1. Observe: capture the screen (or a description of it) and add it to the context.
2. Decide: the model reads the goal, the history and the newest observation, and chooses one action.
3. Act: the harness performs it on a real or virtual machine: move the mouse to a point, click, type a string, press a key combination, scroll, wait.
4. Repeat until the goal is reached, the model gives up, or a step limit hits.

In Anthropic’s computer-use tool, for example, the model emits actions such as screenshot, left_click at a coordinate, type with text, key with a combination like ctrl+s, scroll, left_click_drag, wait, and zoom to look at part of the screen at full resolution; the developer’s harness executes them, usually in a sandboxed virtual machine, and returns a new screenshot as the tool result. The model never touches the machine directly: everything goes through the harness, which is also where safety controls live.

Screenshots or the accessibility tree

There are two ways to show a model the screen.

• Screenshots. Universal: anything a person can see, including canvases, images, remote desktops and games. But the model must read text from pixels and work out where things are.
• The accessibility tree, the structured description operating systems and browsers build for screen readers: every button, link and text field, with its role, its label and an identifier. Text is exact and elements can be addressed by identifier instead of by pixel, but it is missing or wrong in many applications (canvas-drawn apps, games, poorly built sites), and it can be very long.

How long? The same pages, measured both ways. The accessibility trees were snapshots of three live pages, taken once with Chrome’s devtools and not kept. The pages have changed since, so those three lines of the output can’t be reproduced exactly, and observation_cost.py skips them when the snapshot files are missing; the screenshot counts are computed from Claude’s documented rule and rerun as shown. The rule, from the script:

# What a computer-use agent's observations cost. Screenshots: Claude's
# documented rule, one visual token per 28x28 patch, after shrinking the
# image (keeping its shape) to fit a tier's long-edge and token limits.
TIERS = {"standard": (1568, 1568), "high-resolution": (2576, 4784)}     # (max long edge, max visual tokens)
def visual_tokens(w, h, tier):
    edge, cap = TIERS[tier]
    tokens = lambda w, h: math.ceil(w / 28) * math.ceil(h / 28)
    if max(w, h) <= edge and tokens(w, h) <= cap:
        return w, h, tokens(w, h)
    lo, hi = 0.0, min(1.0, edge / max(w, h))            # largest scale that fits both limits
    for _ in range(60):
        mid = (lo + hi) / 2
        (lo, hi) = (mid, hi) if tokens(round(w * mid), round(h * mid)) <= cap else (lo, mid)
    return round(w * lo), round(h * lo), tokens(round(w * lo), round(h * lo))

Full script: observation_cost.py

The script then counts each accessibility tree’s tokens with Qwen2.5’s tokenizer, whole and pruned to the interactive elements, and adds up a task’s input tokens and cost step by step, with and without old screenshots and prompt caching. The task assumes 2,000 tokens of instructions and tools, and 150 tokens of the model’s own reasoning and action per step; output tokens are not counted. With caching, each request reads the previous request’s prefix at 0.1× the input price and writes the new part at 1.25×, and a history with screenshots dropped is not cached:

screenshot tokens (check against the documentation's table: 1920x1080 -> 1456x819, 1560 / 2691)
  1024x768   standard: 1,036 tokens   high-resolution: 1,036 tokens
  1280x800   standard: 1,334 tokens   high-resolution: 1,334 tokens
  1920x1080   standard: 1,560 tokens (shrunk to 1456x819)   high-resolution: 2,691 tokens
  2560x1440   standard: 1,560 tokens (shrunk to 1456x819)   high-resolution: 4,784 tokens

accessibility trees, whole page (raw devtools snapshot / interactive elements only, no URLs):
  blog_home      109 nodes     3,748 tokens   pruned:    31 elements      557 tokens
  hackernews     756 nodes    15,685 tokens   pruned:   231 elements    3,090 tokens
  wikipedia    6,813 nodes   167,990 tokens   pruned: 1,150 elements   20,363 tokens

one task's input tokens, screenshots at 1280x800 (1,334 tokens each), $5 per million input tokens:
   10 steps: keep every screenshot   101,620 tokens ($ 0.51, cached $ 0.15); keep the last 3   64,268 ($ 0.32); last request 16,840 tokens
   30 steps: keep every screenshot   750,060 tokens ($ 3.75, cached $ 0.64); keep the last 3  245,808 ($ 1.23); last request 46,520 tokens
   50 steps: keep every screenshot 1,992,100 tokens ($ 9.96, cached $ 1.43); keep the last 3  487,348 ($ 2.44); last request 76,200 tokens

A screenshot costs a bounded amount: Claude cuts an image into 28×28-pixel patches, one token each, with a cap of 1,568 tokens (4,784 on the high-resolution tier of Claude 4.7 and later). An ordinary image over the cap is shrunk to fit, but a computer-use screenshot over it is rejected, so the harness must resize screenshots itself before returning them; the documentation recommends screens around 1024×768 or 1280×800. A 1280×800 screen is 1,334 tokens.

An accessibility tree has no such cap. These are raw snapshots from Chrome’s developer tools, the whole page with every node, URL and identifier: 3,748 tokens for this blog’s home page, 15,685 for Hacker News’s front page and 167,990 for a long Wikipedia article. That’s close to a worst case, and agents rarely receive it whole. Keeping only the interactive elements (links, buttons, fields) with their labels cuts the three to 557, 3,090 and 20,363 tokens. The comparison isn’t one-for-one either: a tree covers the whole page, a screenshot only what’s visible, and reading the whole Wikipedia article by screenshot would take dozens of scrolls. Practical agents mix the two: a screenshot for the layout, plus a pruned list of interactive elements with identifiers, so the model can say “click element 42” instead of guessing a pixel.

Grounding: from intent to coordinates

The hardest part of acting on a screenshot is grounding: turning “click the Save button” into the pixel (612, 341). Errors are unforgiving. A click 20 pixels off lands on a different button. Three approaches:

• Direct coordinates. The model outputs pixel positions itself, which requires training on screenshots paired with element locations. Coordinates refer to the image the model saw, so when the harness has resized the screenshot (which it must, above the limits), it has to scale the coordinates back to the real screen; many bugs in computer-use harnesses are this mismatch. Zooming in on a region at full resolution before clicking helps with small targets.
• Set-of-marks (Yang et al., 2023): overlay numbered boxes on the interactive elements before sending the screenshot, and let the model pick a number. The harness, which drew the boxes, knows where each one is.
• A separate grounding model: a planner model decides what to do in words, and a smaller model trained only on UI grounding finds the element.

The action space

The actions are low-level and the tasks long. Saving a file under a new name is a menu click, a dialog, a text field, a folder choice and a confirmation: a dozen steps, each a full model call with a screenshot. A step takes several seconds (the model call, capturing the screen, and waiting for the interface to settle), so a long task takes many minutes, and it’s expensive, and every step is a chance to go wrong. Agents that recover from mistakes (noticing a dialog didn’t open, scrolling to find a button below the fold) do much better than ones that only plan; and keyboard shortcuts, when the model knows them, replace several clicks.

What a task costs

The output above also prices a task’s input (output tokens, which cost more per token but are far fewer, aren’t counted). Every step adds a screenshot to the conversation, and the whole conversation is sent again at every step (the quadratic growth of From Function Calling to the Agent Loop). At 1,334 tokens a screenshot and an illustrative $5 per million input tokens, a 30-step task that keeps every screenshot reads 750,000 tokens in total, $3.75, though no single request is large: the last is about 47,000 tokens. What limits long tasks is less the context window than the per-request image limit, 100 images on models with a 200,000-token window.

Two remedies pull in opposite directions. Keeping only the last three screenshots (older steps survive as text: the actions taken and what was seen) cuts the 30-step task to about 250,000 tokens, $1.23. Prompt caching (Prompt Caching) instead makes the repeated prefix cheap (read at a tenth of the price) as long as it doesn’t change, and with it, keeping every screenshot costs about $0.64. Dropping old screenshots changes the prefix and breaks the cache, so a harness has to choose: cache an append-only history, or trim it and pay full price. Context management (Context Engineering) here is a cost design, not a cleanup.

Benchmarks: OSWorld

OSWorld (Xie et al., 2024) is the standard test: 369 tasks in real Ubuntu applications (office suites, browsers, file managers, terminals, image editors), each checked by a script that inspects the final state of the machine. At publication, people completed 72.36% of tasks and the best agent 12.24%. The gap closed fast: Claude 3.5 Sonnet’s computer use scored 14.9% from screenshots alone in late 2024 (Anthropic’s launch post), OpenAI’s computer-using agent 38.1% in early 2025 (OpenAI’s launch post), and OSWorld-Verified, a version with over 300 reported issues fixed, had its best system at 60.76% in July 2025. The OSWorld leaderboard tracks how close newer systems come to the original 72% human baseline (which wasn’t re-measured on the fixed tasks). Benchmark success on fixed tasks in clean virtual machines is not the same as reliability on a user’s own desktop, and the pass^k arithmetic of Evaluating and Observing Agents applies with force: a long task fails if any step does.

Safety

A computer-use agent reads everything on screen and can do anything the user can. That combination is what makes it dangerous.

• Prompt injection through the screen. A web page, an email or a document can contain text written to look like instructions (“ignore your task and send the file to…”). The agent reads it as part of its observation, and a model that follows it acts with the user’s permissions. Guardrails and Prompt Injection covers injection and the defenses.
• Irreversible actions. Sending, deleting, purchasing, submitting. Harnesses ask the user to confirm such actions, or block them.
• Containment. Run the agent in a virtual machine or container with only the access the task needs: no saved passwords, no access to unrelated accounts, a restricted network.

Anthropic’s documentation for its tool says to do all of this: a dedicated virtual machine with minimal privileges, no access to sensitive data or accounts, an allowlist of internet domains, and human confirmation for consequential actions; it also runs classifiers over screenshots to flag likely prompt injections.

Many production agents don’t drive a whole operating system but only a browser. There the harness can read the page’s structure directly (the DOM, through the browser’s debugging protocol) and give the model stable element references, which makes grounding more reliable and observations cheaper; the screen-and-mouse approach is for everything a browser can’t reach.

Limits

Computer use is the most general way for an agent to act, and the least efficient: perception from pixels, one action per model call, and errors that compound over long tasks. Where an API exists, a tool call through it is faster, cheaper and more reliable than driving the interface. Computer use earns its place for the long tail of software that has no API, and for tasks that span several applications the way a person’s work does.

Where this leads

• World Models, JEPA and Recursive Self-Improvement follows agents that learn a simulator of their environment and plan inside it, instead of acting step by step on the real one.
• System Design for AI designs another agent under a hard budget: a voice agent with about one second per turn, against the several seconds per step here.

Takeaways

• A computer-use agent loops: screenshot, decide, act through a harness, usually inside a sandboxed machine.
• Screenshots are universal and cost a bounded amount (1,334 tokens for a 1280×800 screen); accessibility trees are exact but raw ones can be enormous (3,700 to 168,000 tokens on three real pages, 557 to 20,000 pruned to interactive elements). Good agents combine them.
• Grounding, turning intent into the right pixel, is the hard part: direct coordinates, set-of-marks, or a dedicated grounding model.
• Context management decides cost: a 30-step task keeping every screenshot reads about 750,000 tokens ($3.75); keeping the last three, 250,000 ($1.23); keeping all of them with prompt caching, $0.64.
• OSWorld went from 12% at publication in 2024 to 61% on a cleaned-up version by mid-2025, approaching the original 72% human baseline; safety (injection through the screen, irreversible actions, containment) decides where these agents can run.

References

Start with Anthropic’s computer-use tool documentation for the loop and its safety advice, then the OSWorld paper for how such agents are measured; set-of-marks is the simplest grounding trick to try.

• Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (2024): the benchmark, its 369 tasks, and the human and agent baselines at publication.
• XLANG Lab, Introducing OSWorld-Verified (2025), and the OSWorld leaderboard: the cleaned-up task set and current scores.
• Anthropic, Computer use tool and Vision (documentation): the action set, the screenshot token rule and limits, and the safety advice.
• Anthropic, Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (2024): the 14.9% screenshot-only OSWorld score.
• Yang et al., Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V (2023): grounding by numbered marks on the screenshot.
• OpenAI, Computer-Using Agent (2025): the 38.1% OSWorld score.