Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MNIST-PRO, a benchmark designed to evaluate AI agents' ability to construct and interpret perceptual states in partially observable environments by transforming MNIST digit recognition into a sequential, glimpse-based search task. The study assesses ten multimodal models across various memory representations and reveals significant performance gaps under partial observability, highlighting three key bottlenecks: challenges in integrating fragmented glimpses, premature exploration cessation, and difficulties in revising incorrect beliefs. These findings underscore that effective visual evidence acquisition alone is insufficient; agents must also adeptly construct and update their perceptual states to succeed in dynamic environments.
Agents falter in partially observable settings, revealing that merely gathering visual evidence isn't enough without robust perceptual state management.
AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.