Search papers, labs, and topics across Lattice.
PyroDash introduces a novel framework for cost-efficient collaborative inference between small and large language models, enabling small language models to selectively request assistance from larger models during token generation. This approach integrates a control token mechanism and a Collaborate Engine that facilitates a seamless handoff to a frozen LLM, allowing for enhanced reasoning capabilities without the need for extensive retraining or additional infrastructure. The results demonstrate that PyroDash can improve accuracy by over 6 percentage points while significantly reducing inference costs, showcasing its effectiveness across multiple mathematical reasoning benchmarks.
By intelligently offloading tasks to a large language model, PyroDash achieves a remarkable balance of accuracy and cost, slashing expenses from $49.36 to just $1.78 per example.
Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $\lambda=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $\lambda=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
