Search papers, labs, and topics across Lattice.
The paper introduces Ego, a method for personalizing vision-language models (VLMs) by extracting visual tokens representing specific concepts using the model's internal attention mechanisms. These extracted tokens act as a memory, allowing the VLM to recall and describe personalized concepts in test images without requiring additional training. Ego achieves strong performance gains across single-concept, multi-concept, and video personalization settings while minimizing personalization overhead compared to existing approaches.
Forget retraining: Ego personalizes VLMs on the fly by extracting and leveraging visual tokens that represent specific concepts using the model's internal attention.
AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.
