Search papers, labs, and topics across Lattice.
This paper introduces Efficient Continual Alignment (ECA), a novel exemplar-free approach for Incremental Learning (IL) in Open-ended Image-to-Text Generation (OpenITG), which adapts the alignment module of pre-trained vision-language models (VLMs) to accommodate evolving visual data categories. By implementing mechanisms such as a Mixture of Query (MoQ) module, Fisher Dynamic Expansion (FeDEx), and Dictionary Replay (DR), ECA effectively retains previously learned knowledge while enabling the model to learn new task-specific features. Experimental results on newly constructed IL OpenITG benchmarks reveal that ECA significantly reduces catastrophic forgetting and enhances performance compared to existing methods.
ECA achieves a remarkable balance between learning new tasks and retaining past knowledge, outperforming traditional methods in dynamic visual environments.
Incremental Learning (IL) for Open-ended Image-to-Text Generation (OpenITG) enables models to continuously generate accurate, contextually relevant text for new images while preserving previously acquired knowledge. Unlike prior studies, this paper addresses a more practical scenario in which the predominant category of visual data shifts over time as environments evolve. In this context, we introduce a new notion of continual alignment, which incrementally adapts the alignment module within pre-trained VLMs to preserve high-quality cross-modal representations. Based on this idea, we propose Efficient Continual Alignment (ECA), a novel exemplar-free IL approach for OpenITG. The key challenge is enabling the model to acquire new, task-specific features while minimizing interference with the established alignment without accessing raw data from previous tasks. To address this, ECA employs three core mechanisms: a Mixture of Query (MoQ) module that adapts task-specific query tokens, a Fisher Dynamic Expansion (FeDEx) that dynamically expands model structure based on a Fisher Information Matrix (FIM)-based metric, and an embedding dictionary with Dictionary Replay (DR) to retain past knowledge. To evaluate ECA's performance, we construct four new IL OpenITG benchmarks that better reflect real-world scenarios. Experimental results demonstrate that ECA significantly mitigates catastrophic forgetting and improves IL performance compared to baseline methods. Code and benchmarks are available at https://github.com/Snowball0823/ECA.
