Search papers, labs, and topics across Lattice.
The paper introduces SELECT TO THINK (S2T), a novel distillation method that leverages the "local sufficiency" property, where a small language model's (SLM) top-K predictions often contain the large language model's (LLM) preferred token at reasoning divergence points. S2T reframes LLM supervision as selection among the SLM's proposals, simplifying the distillation process. S2T-LOCAL distills this selection logic into the SLM, enabling autonomous re-ranking and achieving a 24.1% performance improvement, matching the efficacy of 8-path self-consistency with single-trajectory efficiency.
SLMs can match the reasoning performance of much larger models by simply re-ranking their own top-K token predictions, eliminating the need for expensive LLM calls at inference time.
Small language models (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power exhibited by their larger counterparts (LLMs). To mitigate this gap, current approaches invoke an LLM to generate tokens at points of reasoning divergence, but these external calls introduce substantial latency and costs. Alternatively, standard distillation is often hindered by the capacity limitation, as SLMs struggle to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose SELECT TO THINK (S2T), which reframes the LLM's role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-LOCAL, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, we demonstrate that a 1.5B SLM's top-8 candidates capture the 32B LLM's choice with 95% hit rate. Translating this potential into performance, S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks, effectively matching the efficacy of 8-path self-consistency while operating with single-trajectory efficiency.
