Reading a page the way a person does
5 min read
There are two obvious ways to show an agent a web page, and both of them are wrong on their own.
A screenshot is what the page looks like, which is the right question and an expensive answer. The DOM is what the page is, which is cheap and far too much of it. Neither is what a person uses.
The cost of looking
A full-page screenshot at a readable resolution is on the order of a thousand tokens before the model has been asked anything. Do that on every step of a twenty-step task and the picture costs more than the thinking.
The DOM is worse in the other direction. A modern application page is comfortably a megabyte of markup, nearly all of it layout scaffolding, and the twelve things a person can act on are somewhere inside it.
The third thing
What a person reads is neither: it is the handful of elements that are visible, interactive and labelled. That list is short enough to print, and it is what we hand an agent by default.
- Visible — in the viewport, not behind a dialogue, not zero-sized.
- Interactive — something a click, a keystroke or a scroll would change.
- Labelled — by its own text, its accessible name, or the text that sits with it.
Each one gets a number, and the number is what an action refers to. The agent says “click 7” rather than describing a selector it cannot see the result of.
When to look anyway
The list is not enough for everything, and pretending otherwise is how an agent ends up confidently wrong. A chart, a canvas, a map, a layout that has quietly broken — none of them are in it.
So the picture stays available and is taken when it is asked for. The change is that it is asked for a few times in a task instead of on every step, which is the difference between an expensive habit and a tool.