Search papers, labs, and topics across Lattice.
This paper introduces CARE (Confidence-Adaptive Routing of Experts), a novel method for routing tokens in Mixture-of-Experts (MoE) Low-Rank Adaptation (LoRA) models based on per-token uncertainty. By leveraging the router's output distribution to adaptively select experts, CARE improves the allocation of computational resources, ensuring that challenging tokens receive appropriate attention while reducing the activation of experts for simpler tokens. Experimental results demonstrate that CARE outperforms fixed top-k MoE-LoRA configurations across various benchmarks while maintaining computational efficiency and enhancing out-of-distribution detection capabilities.
Activating experts based on token uncertainty allows CARE to outperform traditional fixed routing methods while using fewer resources.
Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$. Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serves hard ones. We observe that the router's output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity. We introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion. Experts are activated in decreasing router weight until their cumulative mass reaches a threshold, with a small extension when the admitted experts disagree. A budget thermostat calibrates the threshold so that the average number of active experts matches any target. CARE is a drop-in, single-forward-pass rule with no extra parameters. Across eight commonsense benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, as well as math, code, and knowledge tasks, CARE improves over fixed top-k MoE-LoRA at matched compute and matches the fixed-k=4 baseline while activating fewer experts. The same confidence and disagreement signals also improve out-of-distribution detection over MSP, entropy, and multi-pass proxies. We support the design with nucleus fidelity, budget optimality, and an epistemic reading of disagreement, and we release code.
UW