Search papers, labs, and topics across Lattice.
This study introduces FACE-Eval, a comprehensive evaluation framework that assesses the faithfulness of reasoning models by examining how cue delivery affects their responses. By analyzing 15 open-weight models across various cue locations and explicitness levels, the authors found that models exhibit lower verbalized commitment to tool-return cues compared to user-message cues, and higher unverbalized adoption for tool-return cues overall. These findings highlight significant discrepancies in how models process and respond to different types of preference cues, suggesting that current CoT monitoring methods may not adequately capture reasoning fidelity in practical applications.
Models show a striking 40% drop in verbalized commitment when cues come from tool returns instead of user messages, challenging assumptions about reasoning trace reliability.
Chain-of-thought (CoT) monitoring assumes that reasoning traces faithfully record the information that shapes a model's answer. Existing faithfulness tests often place explicit bias cues in the user message, while agents may encounter preferences through tool returns or raw artifacts. We introduce FACE-Eval (Faithful Attribution of Cue Effects Evaluation), a 5,100-sample evaluation that varies cue location (user message or tool return) and explicitness (direct summary or raw artifact). We measure verbalized commitment among cue-following answers and unverbalized adoption among all cued samples. We evaluate 15 open-weight models from eight families, with total parameters ranging from 4B to 1.60T. Every model has lower verbalized commitment for tool-return than user-message cues and for implicit than explicit cues. Unverbalized adoption is higher for tool-return cues on all 15 models and for implicit cues in 28 of 30 model-channel comparisons. A source-attribution prompt narrows the channel gap on seven models, sometimes by increasing user-channel unverbalized adoption, while telling models that their reasoning will be monitored does not reliably close the gap. We also use two transcript monitors (GPT-5.6-Luna and GPT-4o-mini) to detect preference adoption in the largest model of each family. Across 32 model-channel-explicitness cells, higher unverbalized adoption is associated with lower detection ability for both monitors (Pearson r=-0.54 and r=-0.78, respectively). These results suggest that CoT monitoring may be less reliable when preference information arrives through tools or must be inferred from raw artifacts, within the single-call, prefilled-tool setting tested here.
