Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces "Trace as State," a novel approach that leverages reasoning traces as a conditional state update for long-context transformers, allowing for more efficient memory usage during processing. By positioning these traces before the long-context block, the method enables previously derived information to inform subsequent reasoning, significantly enhancing performance. Experimental results demonstrate that Trace as State outperforms a control method, Trace Append, across 26 out of 27 combinations of models, tasks, and metrics, achieving remarkable improvements in accuracy on long-context datasets.
Placing reasoning traces before context in transformers can boost long-context reasoning accuracy by up to 100% while using exponentially less memory.
Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update tasks. For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. Motivated by this principle, we introduce Trace as State. We use collected reasoning traces as a textual proxy for task state and place it before the long-context block on a fresh pass, allowing information derived previously to guide rereading. We conduct extensive experiments on Trace as State and Trace Append, a matched control that uses the same task state proxy but put it after the context. Across three models and three long-context datasets, Trace as State outperforms Trace Append in 26 of 27 reported combinations of model, task, and metric. On GraphWalks Parents, exact match lifts DeepSeek V4 Pro Preview from 29.2% on the initial pass and 43.0% with Trace Appendto 81.8% with Trace as State, and from 66.4% and 83.2% to 100.0% for GLM-5.2. These results show that placing traces before the context can improve long-context reasoning while retaining the causal transformer structure.
Tsinghua AI