Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces B^3D-RWKV, a diffusion RWKV model that combines the linear-time inference efficiency of RWKV with the parallel, bidirectional capabilities of discrete diffusion models. The key innovation is a triplet-block layout that enables bidirectional attention within the diffusion process while maintaining the causal structure required for language modeling. Experiments show that B^3D-RWKV-7.2B achieves comparable accuracy to existing models on an 8-task suite while providing a 1.6x speedup in decoding throughput.
RWKV models can now do diffusion, unlocking 1.6x faster decoding without sacrificing accuracy.
Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose $B^3D-RWKV$, a diffusion RWKV variant that integrates the model's $O(L)$ inference efficiency with parallel, bidirectional discrete-diffusion through a \emph{triplet-block layout} method. $B^3D-RWKV-7.2B$ reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of $\mathbf{1.6\times}$ speedup.