Search papers, labs, and topics across Lattice.
The paper introduces SEVEX, an automated semantic exploration framework, to discover task-specific visual prompts that improve Large Vision-Language Model (LVLM) perception. SEVEX addresses the challenges of lengthy code and vast search spaces by using an abstract idea space, novelty-guided selection, and semantic feedback-driven ideation. Experiments on BlindTest and BLINK benchmarks show SEVEX significantly outperforms baselines in task accuracy, inference efficiency, and exploration stability by discovering sophisticated visual strategies.
Forget hand-crafted visual prompts – this framework automatically discovers counter-intuitive image manipulation strategies that dramatically boost LVLM perception.
LVLMs encounter significant challenges in image understanding and visual reasoning, leading to critical perception failures. Visual prompts, which incorporate image manipulation code, have shown promising potential in mitigating these issues. While emerged as a promising direction, previous methods for visual prompt generation have focused on tool selection rather than diagnosing and mitigating the root causes of LVLM perception failures. Because of the opacity and unpredictability of LVLMs, optimal visual prompts must be discovered through empirical experiments, which have relied on manual human trial-and-error. We propose an automated semantic exploration framework for discovering task-wise visual prompts. Our approach enables diverse yet efficient exploration through agent-driven experiments, minimizing human intervention and avoiding the inefficiency of per-sample generation. We introduce a semantic exploration algorithm named SEVEX, which addresses two major challenges of visual prompt exploration: (1) the distraction caused by lengthy, low-level code and (2) the vast, unstructured search space of visual prompts. Specifically, our method leverages an abstract idea space as a search space, a novelty-guided selection algorithm, and a semantic feedback-driven ideation process to efficiently explore diverse visual prompts based on empirical results. We evaluate SEVEX on the BlindTest and BLINK benchmarks, which are designed to assess LVLM perception. Experimental results demonstrate that SEVEX significantly outperforms baseline methods in task accuracy, inference efficiency, exploration efficiency, and exploration stability. Notably, our framework discovers sophisticated and counter-intuitive visual strategies that go beyond conventional tool usage, offering a new paradigm for enhancing LVLM perception through automated, task-wise visual prompts.
