Search papers, labs, and topics across Lattice.
This paper introduces MED-DSLC, a method that enhances multi-expert-domain classification in vision-language models by addressing the challenges of domain supervision and logit calibration. By merging independently trained LoRAs and applying domain-wise logit scaling, the approach effectively restores global logit comparability, which is crucial for accurate zero-shot recognition across diverse domains. Experimental results reveal a significant mean accuracy improvement of 15%, alongside enhanced cross-domain robustness and scalability, highlighting the importance of calibrated outputs in multi-domain settings.
Restoring global logit comparability in vision-language models boosts zero-shot classification accuracy by 15% while maintaining robustness across diverse domains.
Vision-language models (VLMs) such as CLIP enable zero-shot classification by comparing image features with text prompts in a shared embedding space. A fundamental property underlying this capability is the global comparability of logits across arbitrary candidate classes. However, VLMs are often adapted to fine-grained domains using techniques such as LoRA. While this improves in-domain accuracy, out-of-domain accuracy degrades. This leads to a highly fragmented model ecosystem, with thousands of specialized models. Multi-Expert-Domain classification seeks to address this problem, by merging LoRAs trained independently on specialized domains. However, due to the independent training, the various domain experts no longer produce globally calibrated logits. As a result, when evaluating over the union of multiple domain-specific class sets, heterogeneous logit scales induce cross-domain interference and artificially high confidence for out-of-domain classes, inducing prediction errors. In this work, we identify domain supervision and cross-domain logit miscalibration as the key issue to scalable multi-domain zero-shot recognition. We propose MED-DSLC, combining domain supervised training and domain-wise logit scaling, to explicitly restore global logit comparability. MED-DSLC is a lightweight solution for MED classification, which is shown to preserve within-domain discrimination while reducing cross-domain logit interference with minimal data. Extensive experiments across diverse fine-grained benchmarks demonstrate that it substantially improves mean accuracy (+15\%), cross-domain robustness, and scalability in the size of MED classification problem. Our results show that restoring output-level calibration is essential under highly data imbalanced settings for achieving a truly zero-shot VLM under multi-domain specialization.
