Search papers, labs, and topics across Lattice.
This paper introduces Rosetta, a composable native multimodal pretraining framework that addresses the challenges of catastrophic forgetting in existing architectures when integrating new modalities. By employing a modular approach with global shared experts and plug-and-play modality-specific experts, Rosetta preserves foundational knowledge while allowing for seamless expansion. The proposed Momentum-Anchored Orthogonal Projection (MAOP) effectively mitigates gradient conflicts, resulting in robust performance across language and visual tasks without sacrificing previously learned information.
Rosetta achieves multimodal integration without catastrophic forgetting, enabling models to expand their capabilities while retaining prior knowledge.
Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. However, accommodating continuous generative objectives alongside discrete understanding tasks causes severe gradient conflicts. Existing architectures, including standard Mixture-of-Experts (MoE), are highly susceptible to representation overwriting. Even structurally partitioned paradigms like Mixture-of-Transformers (MoT) remain vulnerable to catastrophic forgetting, severely impeding multimodal scalability. In this work, we introduce Rosetta, a composable native multimodal pretraining framework designed for seamless and non-destructive modality expansion. Rosetta adopts a modular paradigm where core foundational knowledge is preserved within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts. To guarantee non-destructive composition, we propose Momentum-Anchored Orthogonal Projection (MAOP). MAOP leverages the optimizer's momentum state as an implicit semantic anchor, selectively neutralizing conflicting gradient components from new modalities while preserving synergistic updates. Extensive evaluations demonstrate that, while standard MoE and MoT architectures suffer catastrophic forgetting of previously acquired knowledge, Rosetta robustly preserves established language and visual understanding. Furthermore, it delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models. To facilitate further multimodal research, we release our code and checkpoints to the community. Project page at https://rosetta-lmm.github.io/.
