Search papers, labs, and topics across Lattice.
BoxTuning addresses the limitations of text-coordinate methods in video question answering by directly injecting object spatial-temporal information into the visual modality of MLLMs. This is achieved by rendering colored bounding boxes and trajectory trails onto video frames, paired with a concise color-to-object legend in the text. Experiments on five video QA benchmarks demonstrate that BoxTuning outperforms text-coordinate baselines, especially on spatially oriented tasks, while also mitigating accuracy degradation on reasoning-centric tasks.
Injecting bounding box information directly into the visual modality of video LLMs slashes token costs by up to 93% and unlocks better spatial reasoning.
Object-level spatial-temporal understanding is essential for video question answering, yet existing multimodal large language models (MLLMs) encode frames holistically and lack explicit mechanisms for fine-grained object grounding. Recent work addresses this by serializing bounding box coordinates as text tokens, but this text-coordinate paradigm suffers from a fundamental modality mismatch: object information is inherently visual, yet encoding it as text incurs a high token cost that forces aggressive temporal downsampling. We propose BoxTuning, which resolves this mismatch by injecting object spatial-temporal information directly into the visual modality. Colored bounding boxes and trajectory trails are rendered onto video frames as visual prompts, with only a concise color-to-object legend retained as text. This reduces the token cost significantly, achieving 87-93% text token reduction in practice. It also preserves full temporal resolution, where the trajectory trails further encode inter-frame motion direction and speed within each keyframe, recovering fine-grained dynamics that text-coordinate methods are forced to discard. Experimental results on five video QA benchmarks (CLEVRER, Perception Test, STAR, NExT-QA, IntentQA) show that BoxTuning surpasses text-coordinate baselines on spatially oriented tasks and nearly eliminates the accuracy degradation observed on reasoning-centric tasks, establishing visual prompting as a more natural and efficient paradigm for conveying object information to video MLLMs.
