Search papers, labs, and topics across Lattice.
This paper introduces TriggerBench, a novel benchmark designed to evaluate prospective memory (PM) in large language models (LLMs) across various real-world scenarios. The authors find that PM is significantly more challenging than retrospective memory (RM), with models exhibiting a precision-recall trade-off and a notable decline in accuracy under implicit constraints. Additionally, the study reveals that PM can serve as an indicator of a model's spare reasoning capacity, demonstrating that successful problem-solving trajectories correlate with higher PM accuracy.
Prospective memory in LLMs is not just harder than retrospective memory; it reveals critical insights into a model's reasoning capacity and attentional robustness.
While Large Language Models (LLMs) are increasingly deployed in long interactions, existing evaluations focus predominantly on retrospective memory (RM) via explicit queries. Prospective memory (PM), the critical ability to spontaneously recall and act on latent constraints without direct prompts, remains largely unevaluated. We introduce TriggerBench, a comprehensive PM benchmark spanning five dimensions across both daily assistants and professional workflows. TriggerBench pairs scenarios with matched RM controls, contrastive positive/negative variants, and overloaded triggers, enabling fine-grained measurement of proactive recall, false-alarm rate, and attentional robustness under a single protocol. Our evaluation yields three key findings. (i) PM shows a precision-recall trade-off and attentional fragility. Though enhanced reasoning significantly improves proactive recall, models may overfit to an "always-remind" heuristic. Furthermore, PM accuracy degrades substantially under implicit constraints or triggers overloaded by concurrent user requests, indicating that robust PM remains an open challenge. (ii) PM is notably harder than RM: on identical contexts, RM near-saturates up to 100K tokens, while PM decays sharply as context length scales. (iii) PM may serve as a behavioral probe of spare reasoning capacity. Pairing PM scenarios with AIME-2025 math problems reveals that successful trajectories yield higher PM accuracy than failed ones at the same context length, showing PM tracks spare reasoning budget that token count obscures. Project page: https://github.com/KristenZHANG/TriggerBench-Official.
Microsoft Research