Search papers, labs, and topics across Lattice.
This paper addresses the issue of Modal Isolation in interleaved thinking, where multimodal models struggle to maintain coherence between text and visual outputs during complex reasoning tasks. By introducing MoTiF, a two-stage training framework that focuses on optimizing modality transitions through Reflective SFT and Flow-GRPO, the authors quantify and mitigate cross-modal hallucination and visual utilization deficits. The results across four visual puzzle benchmarks reveal that enhancing transition-level fidelity significantly improves both cross-modal coherence and overall task accuracy, underscoring the necessity of structural supervision at modality boundaries.
Transition-level supervision can dramatically enhance multimodal model performance, revealing that coherence between text and visuals is crucial for complex reasoning tasks.
Interleaved thinking, where a unified multimodal model alternates between textual reasoning and visual generation, has shown promise on spatial and physical tasks. However, in complex long-chain scenarios, we identify a fundamental failure mode: generated images diverge from the textual context while subsequent text ignores the visual evidence, causing the two modalities to alternate without genuinely informing each other. We term this Modal Isolation and attribute it to compounding information loss at modality boundaries. We decompose each reasoning cycle into atomic operations and define modality transition loss, quantifying cross-modal hallucination (text-to-image) and visual utilization deficit (image-to-text) at each boundary. We propose MoTiF (Modality Tiransition Fidelity), a two-stage training framework that directly optimizes these transitions: Reflective SFT trains the model to detect and recover from erroneous visual outputs; Flow-GRPO improves image generation fidelity via reinforcement learning. All training signals in MoTiF derive from transition-level fidelity rather than end-task accuracy. Across four visual puzzle benchmarks, this transition-level supervision substantially improves both cross-modal coherence and final task accuracy. The results demonstrate that effective interleaved reasoning requires explicit structural supervision at modality boundaries, not merely scaling or end-task optimization.
