Search papers, labs, and topics across Lattice.
This paper introduces TACO, a framework designed to enhance open-vocabulary video recognition by addressing the inconsistency between fine-tuning and evaluation objectives. By employing Relative Structure Distillation, TACO preserves out-of-distribution (OOD) alignment and mitigates harmful shifts in representation during training. The method achieves state-of-the-art performance across various benchmarks, demonstrating its effectiveness in balancing generalization and specialization in video recognition tasks.
TACO achieves state-of-the-art performance in open-vocabulary video recognition by preserving out-of-distribution alignment, challenging the conventional trade-off between generalization and specialization.
Adapting CLIP for open-vocabulary video recognition necessitates a delicate balance between newly acquired video knowledge and the pretrained generalization. While existing studies pursue this generalization-specialization trade-off with additional regularizations or constraints, we argue that they overlook the deviation of representations beyond the fine-tuning data distribution, resulting in suboptimal adaptation effects. We believe such deviation is inherited from the inconsistency between the fine-tuning and evaluation objectives, where model optimization is restricted to the known training distribution but evaluated on unseen ones. In this paper, we introduce \emph{TACO}, a simple yet effective framework to mitigate the potential negative effects induced by this inconsistency. Our key insight is that adaptation should preserve OOD-relevant alignment beyond the training distribution. To this end, we propose \emph{Relative Structure Distillation}, which regularizes the relative geometry of the representation space and suppresses harmful alignment shift during training. We further decouple the representation space from the optimization space with a lightweight specialization projection, allowing task-specific adaptation without directly overspecializing the representations used at test time. \emph{TACO} establishes state-of-the-art performance on diverse benchmarks under cross-dataset and base-to-novel settings. Code will be released at https://github.com/ZMHH-H/TACO.
