Search papers, labs, and topics across Lattice.
This paper critiques the limitations of long-horizon benchmarks in evaluating agent performance, arguing that they often mask underlying failures due to context degradation over extended tasks. By introducing the concept of "horizon residual," the authors propose a method to quantitatively assess the difference between actual task success and predictions based on short-task performance. Their findings indicate that without a proper baseline comparison, the reasons for long-horizon failures remain inadequately understood, highlighting the need for targeted experiments to investigate these failures further.
Long-horizon evaluations reveal that agents suffer from compounding errors, but without proper baseline comparisons, we can't fully grasp the reasons behind their failures.
Long-horizon benchmarks often show that agents fail more as tasks become longer. This observation is useful for deployment, but it does not by itself explain why failure occurs. More stages create more opportunities for ordinary errors to compound; longer tasks may also contain harder individual decisions or become harder as conversation history, tool outputs, and environment changes accumulate. We use trajectory-induced degradation to mean this last possibility: earlier execution makes later work harder. When the harmful accumulation is specifically the text visible to the model, it is often called context rot. In this position paper, we argue that to claim a"long-horizon failure", benchmarks must compare actual full-task success against a baseline prediction built from short, individual stages. We call the log-ratio between this prediction and actual success the horizon residual. The comparison must use the same agent configuration and specify in advance how stages, checkpoints, information, and budgets will be chosen. The residual shows that the full rollout differs from the chosen baseline; targeted experiments are still needed to explain why.
