Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of chain-of-thought (CoT) training in language-model agents by comparing the quality of actions predicted through CoT reasoning versus direct prompt actions. The findings reveal that while prompt actions improve significantly across checkpoints, CoT training does not enhance the relative advantage of CoT reasoning in action selection, indicating a shift towards greater reliance on prompts in later checkpoints. Additionally, selectively masking action-token supervision during training enhances out-of-domain generalization, suggesting a nuanced understanding of CoT's role in agent performance.
CoT training may not boost reasoning capabilities as expected, instead reinforcing reliance on prompts, which could reshape our approach to training language-model agents.
Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning. We therefore ask what CoT training is actually improving: is the model getting better at changing its action through generated reasoning, or is it getting better at predicting the action directly from the prompt? We study this question by comparing \emph{prompt actions} (predicting action without CoT) with CoT actions (predicting action with CoT). Across checkpoints, prompt-action quality improves substantially. While interacting with the environment, the relative advantage of CoT actions over prompt actions remains similar, showing that CoT training does not widen the advantage of CoT reasoning, and it helps to improve the quality of prompt actions. We further find that later checkpoints are less likely to revise the action in response to CoT, suggesting greater reliance on the prompt. Motivated by these patterns, we selectively mask action-token supervision on a fraction of training examples. This intervention improves out-of-domain generalization.
