Search papers, labs, and topics across Lattice.
This paper introduces SemReWrite, a novel framework designed to address evolving semantic concept shifts in visual foundation models by selectively updating outdated visual-semantic mappings while preserving valid knowledge. The authors evaluate their approach using EvoShift-Bench, which encompasses various datasets and semantic transitions, demonstrating that SemReWrite outperforms traditional methods in balancing the learning of new semantics with the retention of unaffected knowledge. Key results indicate that SemReWrite achieves superior performance in terms of Rewrite Accuracy and Preservation Accuracy, highlighting its effectiveness in managing semantic evolution in long-lived visual systems.
SemReWrite achieves a remarkable balance between updating visual-semantic mappings and retaining critical knowledge, outperforming conventional adaptation methods in evolving contexts.
Visual foundation models are commonly adapted under the assumption that the appearance of incoming data may change while the semantic meaning of the prediction task remains fixed. In long-lived visual systems, however, taxonomies, policies, and concept definitions can themselves evolve, causing the same visual evidence to require a different interpretation. We study this setting as evolving semantic concept shift and introduce SemReWrite, a framework for selectively updating obsolete visual--semantic mappings while preserving knowledge that remains valid. SemReWrite represents changes between old and revised semantic specifications, combines semantic discrepancy with sparse revised supervision to localize affected visual regions, and uses an input-dependent low-rank rewriting mechanism together with structured semantic memory, preservation, and obsolete-decision suppression. We further introduce EvoShift-Bench, spanning ImageNet, iNaturalist, CUB-200-2011, and DomainNet, with semantic transitions including class split, merge, boundary revision, insertion, partial redefinition, recurrence, and mixed semantic--appearance shift. To explicitly evaluate selective semantic revision, we introduce Rewrite Accuracy (RA) and Preservation Accuracy (PA) for affected and unaffected regions, respectively, Obsolete Retention (OR) for measuring residual outdated semantic associations, and the Selective Revision Score (SRS), which jointly summarizes rewriting and preservation performance. Experiments show that SemReWrite achieves a stronger balance between learning revised semantics and retaining unaffected knowledge than prompt replacement, conventional fine-tuning, parameter-efficient adaptation, and continual-learning strategies.
