Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of Chain-of-Thought (CoT) prompting in modern mid-sized language models on math problem-solving tasks, revealing that these models often generate CoT-style responses natively. The researchers found that specialized reasoning models outperformed traditional few-shot CoT prompting in a zero-shot setting, achieving accuracy improvements from approximately 77% to 84% on the GSM8K benchmark. This performance enhancement is attributed to a 'guidance-distraction' tradeoff, where standard CoT prompting can hinder rather than help reasoning as models become more capable.
Standard CoT prompting may distract advanced LLMs from their reasoning tasks, leading to worse performance than zero-shot approaches.
Chain-of-Thought (CoT) prompting remains the standard baseline for evaluating models' reasoning abilities. Originally, this technique was introduced to elicit step-by-step reasoning from large language models (LLMs), which would otherwise tend to directly output the final answer. However, many modern LLMs produce CoT-style responses \textit{natively} when presented with reasoning tasks, which made us revisit the effectiveness of standard CoT prompting. We evaluate several modern mid-sized language models on a math problem-solving task and find that models specialized for reasoning achieve better performance in a simple zero-shot setting than when using few-shot CoT examples - significantly surpassing officially reported results at no additional cost (e.g., from $\sim$77\% to $\sim$84\% for Mathstral on GSM8K). For the tested general-purpose model, a zero-shot CoT prompt is also sufficient to outperform a few-shot CoT baseline. We attribute this to a `guidance-distraction' tradeoff: standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task. Our findings suggest that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger.
