Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
TokenRouter addresses the challenges of efficiently serving token-level LLM routing by decoupling request-centric programming from model-centric execution, enabling asynchronous dispatch to dedicated subservers for each LLM. Each subserver utilizes a delayed-batching scheduler, optimized via a throughput model, to mitigate step desynchronization and batch admission delays inherent in existing systems. This novel architecture achieves significant throughput gains, ranging from 2.01x to 64.15x, across various routing algorithms and model configurations.
Token-level LLM routing, previously hindered by system inefficiencies, now achieves up to 64x higher throughput with a novel request-centric, model-centric serving architecture.
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.
Tsinghua AI