Search papers, labs, and topics across Lattice.
The paper introduces Ramen, a test-time adaptation framework for vision-language models that addresses performance degradation under mixed-domain shifts by actively selecting relevant samples from past data. Ramen uses domain consistency and prediction balance criteria to retrieve a customized batch of samples, adapting the model using an embedding-gradient cache for efficiency. Experiments on image corruption and domain shift benchmarks demonstrate Ramen's robust and efficient adaptation capabilities in complex scenarios.
Ramen achieves robust test-time adaptation of VLMs in mixed-domain scenarios by selecting the right samples to adapt to, sidestepping the common pitfall of performance degradation when faced with diverse and inconsistent test data.
Pretrained vision-language models such as CLIP exhibit strong zero-shot generalization but remain sensitive to distribution shifts. Test-time adaptation adapts models during inference without access to source data or target labels, offering a practical way to handle such shifts. However, existing methods typically assume that test samples come from a single, consistent domain, while in practice, test data often include samples from mixed domains with distinct characteristics. Consequently, their performance degrades under mixed-domain settings. To address this, we present Ramen, a framework for robust test-time adaptation through active sample selection. For each incoming test sample, Ramen retrieves a customized batch of relevant samples from previously seen data based on two criteria: domain consistency, which ensures that adaptation focuses on data from similar domains, and prediction balance, which mitigates adaptation bias caused by skewed predictions. To improve efficiency, Ramen employs an embedding-gradient cache that stores the embeddings and sample-level gradients of past test images. The stored embeddings are used to retrieve relevant samples, and the corresponding gradients are aggregated for model updates, eliminating the need for any additional forward or backward passes. Our theoretical analysis provides insight into why the proposed adaptation mechanism is effective under mixed-domain shifts. Experiments on multiple image corruption and domain-shift benchmarks demonstrate that Ramen achieves strong and consistent performance, offering robust and efficient adaptation in complex mixed-domain scenarios. Our code is available at https://github.com/baowenxuan/Ramen .
