Search papers, labs, and topics across Lattice.
This paper introduces GAS, a generation-guided training framework that leverages visual generation as auxiliary supervision to enhance representation learning in Multimodal Large Language Models (MLLMs). By employing Next Embedding Prediction (NEP) within a decoupled Mixture-of-Transformers architecture, GAS allows for improved visual understanding without incurring inference overhead, as the auxiliary generation branch is discarded post-training. Experimental results show significant improvements in multimodal understanding, particularly in perception and spatial comprehension, highlighting the effectiveness of integrating generation tasks that require deep cognitive grounding.
Generation-guided training can significantly enhance multimodal understanding in MLLMs without any inference overhead.
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.
