Search papers, labs, and topics across Lattice.
HRBench is introduced as a unified evaluation framework to benchmark thinking-mode switching strategies in hybrid-reasoning LLMs, organizing the design space along switching strategy families (prompt-based, routing, speculative) and training regimes (training-free, SFT, offline/online RL). The framework evaluates these settings across 6 LLMs and 5 reasoning benchmarks, reimplementing 12+ prior methods within the same pipeline to enable controlled comparisons. Results characterize the effectiveness-efficiency trade-offs of different switching strategies, revealing how training, model scale, and task domain influence the preferred strategy.
Stop comparing apples to oranges: HRBench offers the first unified benchmark to fairly evaluate thinking-mode switch strategies in hybrid-reasoning LLMs.
Hybrid-reasoning large language models (LLMs) expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality against inference cost. However, existing methods for adaptive thinking-mode selection are typically evaluated under different models, datasets, and implementation assumptions, making it difficult to compare their practical behavior. We introduce HRBench, a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs. HRBench organizes the design space along two axes: three switching strategy families, prompt-based selection, external routing, and speculative execution, and four training regimes, training-free, SFT, offline and online RL, yielding 12 controlled evaluation settings. We evaluate these settings across 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, while reimplementing 12+ representative prior methods within the same pipeline. Our analysis characterizes how different switching strategies occupy distinct effectiveness-efficiency trade-off regions: prompt-based methods often provide favorable token-accuracy trade-offs, routing methods offer more stable cost reduction, and speculative methods tend to improve accuracy at higher token cost. We further find that training affects strategies differently, and that the preferred strategy varies with model scale and task domain. HRBench provides reference implementations and a unified evaluation platform to support more controlled research on efficient reasoning in hybrid-reasoning LLMs. Our data, code and repository are available at https://github.com/usail-hkust/HRBench.
