Search papers, labs, and topics across Lattice.
Nemotron 3 Super, a 120B parameter (12B active) hybrid Mamba-Attention MoE model, was pre-trained in NVFP4 on 25T tokens and then post-trained with SFT and RL. The model incorporates LatentMoE, a novel MoE architecture optimizing for FLOP and parameter efficiency, and MTP layers for speculative decoding. Results show comparable accuracy to other 120B models on standard benchmarks, but with 2.2x-7.5x higher inference throughput.
Nemotron 3 Super proves you can achieve comparable accuracy to existing 120B models, but with significantly higher inference throughput, by combining Mamba, Attention, and Mixture-of-Experts.
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
AI2
NVIDIA