Search papers, labs, and topics across Lattice.
This paper introduces Test-Time Self-Distillation (TTSD) with Fisher-Anchored Restoration (FAR) to enhance emotion recognition in large video-language models (LVLMs) under conditions of missing or noisy modalities. By employing a frozen teacher model to guide a low-rank adaptive student, the method effectively mitigates the risks of drift and degradation during continual adaptation to partial observations. Experiments demonstrate that TTSD-FAR significantly outperforms existing state-of-the-art methods, maintaining performance stability even with 50% missing modalities across various datasets.
TTSD-FAR maintains consistent performance in emotion recognition even when up to half of the input modalities are missing, outperforming traditional adaptation methods.
Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.
