Search papers, labs, and topics across Lattice.
This study investigates the degradation of routing accuracy in production LLM assistants as the catalog of specialized tools expands, focusing on a dataset of 110 agents and 584 tools. The authors identify two primary sources of degradation: a retrieval gap, where the model fails to surface the correct tool, and a confusion gap, where even optimal retrieval leads to a drop in performance. By employing embedding-based shortlisting, the researchers achieve a recovery of 10-11 percentage points in routing F1 across all models, validated by a human annotation study that shows similar improvements in real-world traffic scenarios.
Routing accuracy in LLM assistants drops significantly as tool catalogs grow, but embedding-based shortlisting can recover up to 17 percentage points in performance.
Production LLM assistants route user requests to growing libraries of specialized tools, but how does routing accuracy degrade as the catalog scales? We study single-step routing on a 110-agent, 584-tool catalog from a deployed enterprise productivity assistant, evaluating three frontier models from 10 to 110 agents. Routing F1 on under-specified requests drops 16--23 percentage points across models. An oracle analysis decomposes the degradation into a \emph{retrieval} gap (the model cannot surface the right tool) and a \emph{confusion} gap (even with perfect retrieval, the oracle ceiling drops 10pp). Embedding-based shortlisting recovers +10--11pp F1 at full scale across all three models and two providers. A production annotation study (1,435 human-labeled utterances, three annotators) confirms the recovery on real traffic at +10--17pp despite 10--15pp lower absolute performance.
