Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of observation representation (HTML vs. accessibility trees) on web agent performance, finding that higher-capability LLMs benefit from verbose HTML observations, especially with larger thinking token budgets, while lower-capability models perform better with compact accessibility trees. Error analysis reveals that higher-capability models leverage layout information in HTML for improved action grounding, whereas lower-capability models are prone to hallucination with longer inputs. The study also demonstrates that incorporating observation history, particularly using diff-based representations, enhances performance across various models and settings.
Counterintuitively, giving web-navigating LLMs *more* HTML to read—not less—unlocks better action grounding and overall performance, but only if the model is capable enough and given sufficient "thinking" tokens.
Web agents based on large language models (LLMs) rely on observations of web pages -- commonly represented as HTML -- as the basis for identifying available actions and planning subsequent steps. Prior work has treated the verbosity of HTML as an obstacle to performance and adopted observation reduction as a standard practice. We revisit this trend and demonstrate that the optimal observation representation depends on model capability and thinking token budget: (1) compact observations (accessibility trees) are preferable for lower-capability models, while detailed observations (HTML) are advantageous for higher-capability models; moreover, increasing thinking tokens further amplifies the benefit of HTML. (2) Our error analysis suggests that higher-capability models exploit layout information in HTML for better action grounding, while lower-capability models suffer from increased hallucination under longer inputs. We also find that incorporating observation history improves performance across most models and settings, and a diff-based representation offers a token-efficient alternative. Based on these findings, we suggest practical guidelines: adaptively select observation representations based on model capability and thinking token budget, and incorporate observation history using diff-based representations.