Search papers, labs, and topics across Lattice.
This paper introduces DocOps, a verifiable evaluation framework designed to assess the capabilities of autonomous agents in manipulating complex digital documents. By systematically evaluating both closed- and open-source models, the authors identify significant limitations in handling intricate, long-range tasks, revealing three critical failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. The findings highlight the current boundaries of agent performance in maintaining document consistency, informing future designs for more robust agents in digital environments.
Even the most advanced autonomous agents struggle with maintaining document consistency, revealing critical failure modes that could hinder their effectiveness in real-world applications.
As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents'manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.
