Search papers, labs, and topics across Lattice.
The paper introduces Mobius-v0, a novel architecture that separates knowledge storage and reasoning processes, utilizing a globally shared Memory for knowledge vectors and multiple Reasoners for compositional reasoning. This architecture allows for improved knowledge compression and reasoning efficiency, as evidenced by a 7B model that matches the performance of a baseline Transformer model while using only 62.6% of the training data. Additionally, the Intern-S2-Mobius model, pre-trained from a larger model, achieves comparable downstream scores with nearly four times the inference speed, highlighting the architecture's practical advantages.
Achieving similar performance to larger models with significantly less data and faster inference speeds could redefine efficiency benchmarks in foundation models.
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.
