Search papers, labs, and topics across Lattice.
The paper investigates why two-stream attention is crucial for the performance of any-order autoregressive models (AO-ARMs) in masked diffusion. It argues that a structural-semantic tradeoff exists in AO-ARMs, where hidden representations need to attend to both semantically informative tokens and structurally recent tokens, requiring specialized attention streams. The authors introduce Decoupled RoPE to isolate this tradeoff from position-content separation, demonstrating that two-stream attention's benefit extends beyond merely decoupling position and content.
Two-stream attention in any-order autoregressive models isn't just about separating position from content, but about resolving a deeper structural-semantic tradeoff that single-stream models can't handle.
Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.
