Search papers, labs, and topics across Lattice.
The paper introduces PITA, a large-scale dataset of propositional logic statements and proofs, to study the length generalization capabilities of reasoning trace (RT) models. They investigate the impact of task depth (proof length) and task breadth (number of unique examples) on RT model performance, comparing them to non-RT baselines. Results show that RT models excel on broad, shallow tasks but struggle with narrow, deep tasks, suggesting inherent limitations in their ability to generalize to longer proofs in constrained problem spaces.
Reasoning traces can actually *hurt* generalization on complex, narrow reasoning tasks, challenging the assumption that they universally improve performance.
Recent years have witnessed meteoric progress in reasoning models: neural networks that generate intermediate reasoning traces (RTs) before producing a final output. Despite the rapid advancement, our understanding of how RTs support reasoning, and the limits of this paradigm, remain incomplete. To promote greater clarity, we introduce PITA: a novel large-scale dataset of over 23 million statements in propositional logic and their corresponding proofs. As a benchmark for robust reasoning, we focus on length generalization: if a model is trained to determine truth or falsity on statements with proofs up to fixed length, how well does it generalize to statements requiring longer proofs? We propose notions of (1) task depth and (2) task breadth, which measure respectively (1) the number of steps required to solve an example from a task and (2) the number of unique examples across a task. We vary these quantities across subsets of PITA, and find that RT models generalize well on broad and shallow subsets, while deteriorating on narrow and deep subsets relative to non-RT baselines. To determine whether our results are idiosyncratic to PITA or indicative of general phenomena, we compare our results to a simple synthetic task based on syllogisms. Our resulting theory suggests fundamental scalings that limit how well RT models perform on deep tasks, and highlights their generalization strengths on broad tasks. Our findings overall identify fundamental benefits and limitations inherent in using reasoning traces.
