Search papers, labs, and topics across Lattice.
This paper tackles the problem of inefficient compute allocation in multi-turn reasoning by framing it as a sequential decision-making problem. They introduce Turn-Adaptive Budgets (TAB), a policy trained with Group Relative Policy Optimization (GRPO) to dynamically allocate tokens across turns, prioritizing harder reasoning steps. Experiments on mathematical reasoning show TAB achieves up to 35% token savings while maintaining accuracy compared to static baselines, and a variant using future sub-question information (TAB All-SubQ) saves up to 40%.
LLMs can save up to 40% of tokens in multi-turn reasoning by adaptively allocating compute based on turn difficulty, without sacrificing accuracy.
As LLM reasoning performance plateau, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing, and difficulty-based budget allocation primarily focus on single-turn settings and fail to address the sequential dependencies inherent in multi-turn reasoning.In this work, we formulate multi-turn reasoning as a sequential compute allocation problem and model it as a multi-objective Markov Decision Process. We propose TAB: Turn-Adaptive Budgets, a budget allocation policy trained via Group Relative Policy Optimization (GRPO) that learns to maximize task accuracy while respecting global per-problem token constraints. Consequently, TAB takes as input the conversation history and learns to adaptively allocate smaller budgets to easier turns and save appropriate number of tokens for the crucial harder reasoning steps. Our experiments on mathematical reasoning benchmarks demonstrate that TAB achieves a superior accuracy-tokens tradeoff saving up to 35% tokens while maintaining accuracy over static and off-the-shelf LLM budget baselines. Further, for systems where a plan of all sub-questions is available apriori, we propose TAB All-SubQ, a budget allocation policy that budgets tokens based on the conversation history and all past and future sub-questions saving up to 40% tokens over baselines.
CMU ML