Search papers, labs, and topics across Lattice.
This paper introduces Self-Prophetic Decoding (SeProD), a novel framework for improving visual search in Large Vision-Language Models (LVLMs) by leveraging self-regulation between pre- and post-training models. SeProD uses probability-based prophetic sampling, where the pre-training model proposes tokens and the post-training model selectively accepts them based on its output distribution, thus mitigating capability deterioration and long-context interference. Experiments demonstrate that SeProD consistently enhances visual search performance across multiple benchmarks and LVLMs without requiring additional training or computational overhead.
LVLMs can now perform visual search far more effectively thanks to a clever decoding strategy that harmonizes pre- and post-training capabilities.
Large Vision-Language Models (LVLMs) are rapidly evolving toward true multimodal reasoning, with visual search representing a concrete instantiation of the thinking-with-images paradigm. However, LVLM visual search faces two key challenges: incompatibility among intrinsic capabilities after post-training, and interference in long multi-step reasoning contexts. To address these, we identify two novel insights. First, self-regulation between pre- and post-training LVLMs leverages the intrinsic single-step capabilities of the pre-training model to mitigate capability deterioration and long-context interference. Second, probability-based prophetic sampling, replacing naive prompting, provides a probabilistic interface where the pre-training model acts as a prophet and the post-training model selectively accepts prophetic tokens under its output distribution, preserving coherent multi-step reasoning. Building on these insights, we introduce SeProD, a self-prophetic decoding framework that leverages intrinsic single-step capabilities to enable coherent multi-step reasoning in a training-free, plug-and-play manner. Experiments show that SeProD consistently improves multiple visual-search LVLMs across all 12 splits of 4 visual search benchmarks, as well as across general VQA benchmarks, without added computational overhead, thanks to its parallel prophetic acceptance mechanism.
