Search papers, labs, and topics across Lattice.
This paper introduces Nemotron-TwoTower, a novel block-wise autoregressive diffusion model that separates context representation and iterative denoising into two distinct towers, enhancing the efficiency of language generation. By employing a frozen autoregressive context tower alongside a trainable diffusion denoiser tower with bidirectional block attention, the model achieves a significant increase in generation throughput while maintaining high quality. Trained on 2.1 trillion tokens, Nemotron-TwoTower retains 98.7% of the performance of traditional autoregressive models while delivering 2.42 times faster generation speeds.
Achieving 2.42X faster generation without sacrificing quality, Nemotron-TwoTower redefines the efficiency of language modeling.
Diffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: a frozen AR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model, and trained on approximately 2.1T tokens, Nemotron-TwoTower retains 98.7% of the autoregressive baseline's quality while offering 2.42X higher wall-clock generation throughput. We release the code and model weights at https://huggingface.co/collections/nvidia/nemotron-twotower.
NVIDIA