9.8 Computer-use and browser agents
You understand what breaks when an agent touches a real UI.
Before:08. Large Language Models
Computer-use agents act on real interfaces by screenshot-and-click or DOM control, and the web fights back: fragile selectors, layout changes, rate limits, CAPTCHAs. Sandboxing and permission scoping are what make the blast radius survivable. It sits late in the module as the hardest surface. The demo scepticism is the takeaway — an agent recorded working on one page degrades across the real web's variety, and reliability numbers only mean something across many sites and days.
Work through these
Screenshot-and-click vs. DOM-based control
Controlling a computer by looking at the screen and clicking, against controlling it through the page's underlying structure. The first is general and fragile; the second is precise and breaks when the page changes.
Selector fragility and recovery
Whichever approach is used, the references it depends on change without warning, and the agent needs a way to notice and recover. Recovery design is most of the engineering here.
Sandboxing and permission scoping
An agent driving a real interface should run in a confined environment with only the permissions the task needs. The blast radius of a mistake is otherwise the whole machine.
Rate limits, CAPTCHAs and terms of service
Sites impose limits, present challenges intended to stop automation, and set terms about it. Respecting these is both a legal and a practical requirement rather than an obstacle to route around.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.