Search papers, labs, and topics across Lattice.
This study introduces an observability ladder to assess how different levels of reasoning visibility in large language models (LLMs) affect users' ability to judge the correctness of responses. By comparing the effectiveness of final responses, self-summaries, full reasoning traces, and internal signals across multiple models and benchmarks, the authors find that while summaries provide some insight, they are less effective than full traces when prompts are withheld. The key result reveals that in scenarios where prompts are known, the trace outperforms summaries in predicting correctness, suggesting that monitorability depends on both the information displayed and the user's context.
Summaries may seem helpful, but they often mislead users about correctness compared to the full reasoning trace, especially when prompts are withheld.
Large language models often show users a final response and a short reasoning summary while the full reasoning trace stays hidden. We introduce an observability ladder that holds each completed run fixed and varies only what a reader inspects to judge whether the answer is correct: the response, a self-summary the model writes from the trace, the trace itself, and internal signals, each with and without the prompt. Across three benchmarks and five open-weight Qwen3 and gpt-oss models, we train matched linear correctness predictors on each access level. Without the prompt, summaries carry most of the trace's ranking signal (mean AUROC 0.774 versus 0.813) and add +0.156 over the response alone. With the prompt visible, the summary's gain collapses to +0.019, while the trace still adds +0.041. Even at equal length, the trace's last words predict correctness as well as summaries, or slightly better, and carry denser and more discriminative uncertainty and self-correction cues. On MMLU-Pro questions with both correct and incorrect runs, linear summary readers are near chance and trace readers retain only modest signal, both with and without the prompt (prompt-withheld AUROC 0.503-0.545 versus 0.544-0.590). With the prompt withheld, a GPT-5-mini reader recovers substantially more signal from both summaries and traces on gpt-oss-20b, and even then the trace keeps a small +0.034 advantage. Much of the linear readers' trace signal is associated with length. In the common case where users already hold the prompt, summaries are less helpful than the full trace for monitoring correctness. Monitorability is thus a joint property of the display and the reader, so any monitorability claim, including for faithfulness, should specify both.
