Search papers, labs, and topics across Lattice.
The SPIRAL framework enhances language model reasoning by integrating sequential, parallel, and aggregative inference methods into a unified pipeline. By employing set reinforcement learning to optimize the generation of independent reasoning traces and standard reinforcement learning for aggregation, SPIRAL significantly improves the efficiency and performance of language models during inference. Experimental results indicate that SPIRAL achieves up to 11 times greater scaling efficiency and a 15% performance boost compared to traditional methods like GRPO when leveraging all three reasoning primitives.
SPIRAL achieves a remarkable 15% performance increase by combining sequential, parallel, and aggregative reasoning in language models.
Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives -- sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL), a framework in which a language model is trained to use all three primitives, as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses. Our experiments on reasoning tasks show that SPIRAL effectively scales with inference compute, outperforming GRPO by up to 11$\times$ scaling efficiency and 15% higher performance when all three compute primitives are scaled.
Stanford HAI