The GUI Agent That Clicked the Right Button on the Wrong Screen
A computer-use agent takes a screenshot, reasons about it, decides to click the "Confirm" button at pixel (840, 612), and dispatches the click. By the time the cursor lands, a modal has appeared. The pixel that was "Confirm" three seconds ago is now "Delete." The agent did exactly what it planned. It planned against a screen that no longer exists.
This is not a grounding error. The model correctly identified the button. It is not a reasoning error. The plan was sound. It is a timing error — the most under-instrumented failure class in GUI automation — and your test suite almost certainly does not cover it, because your test environment never moves between the observation and the action.
The uncomfortable measurement: one recent study of desktop agents on real Ubuntu workloads found a mean gap of 6.51 seconds between when an agent observes the screen and when it acts on that observation. Six and a half seconds is an eternity for a UI. Notifications fire, lazy lists finish loading, animations settle, focus shifts. The agent's mental model of the screen has a shelf life, and almost no agent framework treats it that way.
