Search papers, labs, and topics across Lattice.
This paper introduces SCOPE-Router, a novel dual-tower model routing approach designed for execution-oriented tasks that optimally selects from a diverse pool of visual language models (VLMs) while considering both performance and cost. The authors present VLM-ExecRouterBench, a comprehensive benchmark that evaluates routing across multiple domains, and propose a cost-aware objective (CRM+RCCR) that enhances routing accuracy by effectively managing multi-positive signal dilution. Empirical results demonstrate that SCOPE-Router outperforms existing methods, achieving superior rank scores in both open-set and out-of-distribution evaluations, thereby highlighting its practical advantages in real-world applications.
SCOPE-Router not only outperforms existing VLM routing methods but does so while integrating cost considerations, redefining efficiency in model selection for execution-oriented tasks.
Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost. Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios, and employs training objectives that dilute multi-positive signals via softmax normalization without incorporating cost. We address these limitations with three contributions: (1)VLM-ExecRouterBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains with 11 candidate models spanning nearly two orders of magnitude in pricing; (2)SCOPE-Router, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration (random/diagnostic/diversity sampling), enabling new models to join routing without retraining; (3)CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space. Empirically, SCOPE-Router achieves the best Rank Score on all three benchmarks, surpassing the runner-up by 1.84 points under OOD settings and by 6.75 points under doubly OOD open-set evaluation. When applied to four diverse routers, CRM+RCCR improves Rank Score by 1.25--6.21 points.
