Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of various observation modes for web agents, revealing that while each mode addresses unique task challenges, the optimal choice varies significantly across different task sets. The authors quantify the noise in outcomes, finding that rerunning the same mode can yield a 12-14% variance in results, which diminishes the perceived value of an oracle that selects the best mode for each task. Ultimately, they establish that a cost-effective strategy is to route only unsolved tasks to the cheapest mode, achieving a 9.5-30.6% cost reduction without compromising success rates, while highlighting the limitations of current routing supervision tied to agent performance.
Routing supervision falters precisely when it's most needed, as weaker agents yield fewer labels, limiting the potential for optimal mode selection.
Web agents observe a browser through text, pixels, or both, and the choice is usually fixed once for all tasks. We measure six observation modes across eight site-model combinations (cells) on VisualWebArena and WebArena and ask what choosing per task would buy. The modes are complementary: each solves tasks the others miss, they fail in structurally different ways, and the best choice reverses between task sets. The obvious prize, an oracle that picks a winning mode for every task, looks large but is inflated by run-to-run noise: rerunning the same mode on the same tasks changes 12-14% of outcomes, so a second run of a mode already in hand gains about as much as adding a new one. What survives is a cost bound: sending only the tasks no mode solves to the cheapest mode cuts cost by 9.5-30.6% in 8 of 8 cells at unchanged success. We then test five routing policies (picking the mode, deciding when to spend on the strong mode, a zero-cost rule read off the task text, a confidence cascade, and pooled cost tiers), and none robustly beats simply fixing one well-chosen mode; the one exception is a fragile result in our sparsest cell. The central obstruction is that routing supervision is produced at the agent's success rate: the weaker the agent, the fewer labels a router gets, exactly where routing would be most valuable. This limit belongs to today's agents rather than to routing itself. Label supply and routing opportunity rise together (correlation 0.95 across cells), so a stronger agent can overturn the result, and we report the rerun noise bands and the full measurement protocol.
