Search papers, labs, and topics across Lattice.
This paper introduces Stochastic Turn Depth (StochasT), a novel approach for Visual Instruction Tuning (VIT) that addresses the limitations of existing benchmarks by stochastically grouping language tasks related to the same image into clusters of varying sizes. By preserving the organic order of tasks while mitigating visual attention decay and contextual overfitting, StochasT enhances the multimodal reasoning capabilities of Large Vision-Language Models (LVLMs) in both single-turn and multi-turn scenarios. Experimental results show that StochasT significantly improves LVLM performance, demonstrating its effectiveness in harmonizing capabilities across different interaction contexts.
StochasT reveals that intelligently grouping language tasks can unlock the full potential of LVLMs, enhancing their performance in both single-turn and multi-turn interactions.
Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.
