9.8 Computer-use and browser agents

What this is and why it exists

An agent that drives a real interface is the most impressive demonstration in this module and the least reliable thing in production, because the interface was built for people and changes without warning. This topic is what actually breaks, why recovery matters far more than the happy path, and the confinement that makes a mistake survivable — plus the limits that are not obstacles to be routed around.

The vocabulary

• Computer use — an agent controlling a machine as a person would.
• Screenshot-and-click — acting from an image of the screen and pointer coordinates.
• Structural control — acting through the page's underlying elements.
• Selector — the reference identifying an element to act on.
• Flakiness — a step that succeeds sometimes and fails sometimes.
• Sandbox — a confined environment the agent runs inside.
• Permission scoping — granting only the access the task needs.
• Rate limit — a cap a service places on how often you may call it.

The mental model

Two ways to control an interface, with opposite strengths.

Screenshot-and-click takes an image of the screen, decides where to click or what to type, and acts by coordinates. It is general — it works on anything a person can see, including applications with no interface for automation at all, and that generality is its whole appeal. It is also fragile and expensive: an image per step costs many tokens, a layout change moves the target, scrolling changes coordinates, and rendering differences between machines move things a few pixels. And it is slow, since every step is a full model call over an image.

Structural control acts through the page's underlying elements, addressing them by their attributes. It is precise, fast and cheap — an element reference is exact, actions are immediate, no image is needed — and it only works where such a structure is exposed, which on the web it is and in most desktop applications it is not. Its own fragility is different: a reference that depended on a generated class name or a position in the document breaks the next time the page is rebuilt, and a page that renders after loading may not have the element yet when you look.

Use structural control wherever it is available, because precise and cheap beats general and expensive when both apply, and keep the visual approach for what cannot be reached otherwise. Hybrid designs are common and sensible: structure for navigation and form filling, an image when something unexpected appears and the agent needs to see what.

Then the thing that decides whether any of it works: recovery, not the happy path.

A demonstration recorded on one page on one day is not evidence. The site will change — a banner appears, a field moves, a confirmation step is added, a label is reworded — and every one of those breaks a script written against the previous version. So the design question is not "does it work" but "what does it do when the page is not what it expected", and that is most of the engineering here.

Five habits carry it. Address elements by what they mean, not by where they are — a stable identifier, a label, an accessible role and name, in that order of preference — because those survive a redesign that moves everything. Wait for conditions rather than for time: waiting a fixed two seconds is flaky in both directions, and waiting until the element exists is not. Verify after every action: an agent that clicks and assumes has no idea it is now on the wrong page, so check that the expected thing happened before continuing, which is the single change that most improves reliability. Have a defined behaviour for the unexpected — screenshot, log, attempt one recovery, then stop and ask a person, rather than continuing to act on a page it does not understand. And make failure loud: a task that silently did the wrong thing is much worse than one that stopped and said it could not proceed.

Sandboxing and permission scoping are what make the blast radius survivable, and they are not optional for an agent that acts on a real machine.

Run it in a confined environment — a container or a virtual machine, disposable, with only the files the task needs, so a mistake destroys a sandbox rather than a workstation. Control network egress, allowing only the hosts the task requires, which is what stops a compromise from becoming a data transfer to somewhere unexpected. Use scoped, short-lived credentials rather than a person's own session — an agent logged in as you can do everything you can do, including things nobody intended. Keep irreversible actions behind a gate, as the tools topic requires, and note that in an interface those are frequently one click away from ordinary ones. And record the session, because a visual agent's failures are much easier to understand when you can watch them.

The limits are requirements, not obstacles. Rate limits exist so a service stays available for everybody: respect them, back off when asked, and identify your automation honestly. Terms of service state what a site permits — many prohibit automated access outright, and some permit it under conditions — and reading them is part of the work rather than an afterthought, particularly for anything you would put into production or a portfolio.

Challenges intended to stop automation are a clear signal: the site is saying it does not want automated access here, and the correct response is to stop and find a supported route — an official interface, a data export, an agreement with the operator, or a person doing that step. Building a system that defeats them is a poor engineering decision as well as a legal exposure, because it will be a permanent maintenance burden against an operator who is actively working to prevent it.

And a preference worth stating plainly. If a service offers a programmatic interface, use it. A documented interface is stable, fast, cheap, testable and permitted, and driving its web pages instead is choosing the worst option on every one of those. Computer use is the tool for when there is no interface, which is a real and common situation — legacy internal applications, desktop software, systems nobody will extend — and it is the wrong tool whenever an interface exists.

What you should now be able to explain or do

Compare the two control approaches on generality, precision, cost and speed, and choose per situation. Say why a recorded demonstration is not evidence. Apply the five recovery habits, especially verifying after every action. Confine an agent with a sandbox, egress control and scoped short-lived credentials, and record the session. Treat rate limits and terms as requirements, and stop rather than defeat anti-automation challenges. Prefer a programmatic interface whenever one exists.

Check yourself