Search papers, labs, and topics across Lattice.
This paper introduces TASM (Task-Aware Structured Memory), a novel framework designed to enhance the scalability of multi-modal large language models (MLLMs) by enabling dynamic memory construction that is both task-aware and structure-preserving. By employing task-vector guided compression and semantics-aware token merging, TASM effectively aggregates information without losing semantic integrity, addressing the limitations of existing memory compression methods that often disrupt the underlying structure of visual representations. Evaluations demonstrate that TASM achieves high performance even under significant memory compression, striking a balance between efficiency and adaptability for in-context learning tasks.
TASM achieves high performance in multi-modal tasks while compressing memory by dynamically adapting to task-specific needs without sacrificing semantic integrity.
Multi-modal large language models (MLLMs) depend on in-context learning (ICL) for rapid task adaptation, but their scalability is severely limited by finite context windows and the growing cost of key-value (KV) caches in long multi-modal sequences. Existing memory compression approaches typically rely on rigid token removal or sample-dependent importance estimation, which introduces bias, disrupts semantic structure, particularly for visual representations, and yields static memories that cannot adapt to new queries. We introduce TASM (Task-Aware Structured Memory), a training-free framework that addresses these limitations through task-aware, structure-preserving, and dynamically accessible memory construction. TASM employs task-vector guided compression to replace sample-specific signals with a task-level direction that captures shared relevance across demonstrations. To preserve the underlying manifold, it applies semantics-aware token merging via bipartite graph matching, aggregating tokens without destructive pruning. Finally, TASM structures memory into a hierarchy comprising a compact Core Memory and a Latent Bank, facilitating query-adaptive dynamic retrieval. Evaluations confirm TASM maintains high performance under heavy compression, effectively balancing efficiency with adaptability.
