Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates whether a unimodal language model can provide effective feedback to tune a multimodal vision-language model (VLM). They propose a method where a language agent provides feedback to a VLM to adapt text generation according to the agent's preferences. Experiments demonstrate that LLM preference feedback enhances VLM descriptions, leading to improvements of up to 13% in absolute accuracy and a 64.6% preference alignment rate with human judgments.
A unimodal LLM can effectively guide a multimodal VLM's text generation, suggesting a scalable path to enhance existing LLMs with multimodal capabilities.
To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.