Search papers, labs, and topics across Lattice.
This paper introduces Dynamic Context Adapter (DCA), a novel method for integrating historical context into Vision-Language Models (VLMs) without the computational overhead typically associated with frame concatenation. By utilizing fixed-size, dynamically compressed memory, DCA effectively preserves historical semantics while reducing attention FLOPs by over 25% and achieving 13% memory savings. The approach enhances the performance of VLMs on long-horizon tasks, addressing critical limitations in sequential decision-making applications that require temporal understanding.
Overcoming the quadratic complexity of historical context integration, DCA achieves significant efficiency gains while enhancing performance in Vision-Language Models.
Historical context integration presents a fundamental challenge for Vision-Language Models (VLMs) in sequential decision-making tasks. Current VLMs process visual inputs independently, which creates critical limitations for downstream applications that require temporal understanding. Direct incorporation of historical frames into Transformer inputs produces quadratic attention complexity and excessive memory consumption. Existing approaches suffer from significant drawbacks: computational inflation or substantial information loss through temporal compression. To address these challenges, we introduce Dynamic Context Adapter (DCA), a novel context injection approach for pretrained VLMs. Our method employs fixed-size, dynamically compressed memory to preserve historical semantics without frame concatenation. DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency. DCA achieves over $25\%$ reduction in attention FLOPs and $13\%$ memory savings while improving performance on long-horizon tasks.
