Search papers, labs, and topics across Lattice.
RouteJudge is an innovative online framework designed for evaluating the decision quality of LLM routing systems through pairwise user preference comparisons. Unlike traditional model-level evaluations, it focuses on the effectiveness of routing strategies by allowing users to compare responses from different models under identical constraints. The platform not only captures detailed evaluation records but also introduces ORBIT, a modular toolbox that standardizes the development and evaluation of routing algorithms, enhancing reproducibility and preference-aware analysis in LLM routing.
RouteJudge reveals that user preferences can significantly inform the effectiveness of LLM routing strategies, transforming how we evaluate model performance.
We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/AIGNLAI/LAMDA-ORBIT.
