Search papers, labs, and topics across Lattice.
This paper investigates the distinct mechanisms that arise when training Autoregressive (AR) and Non-Autoregressive (NAR) models, specifically Discrete Diffusion Language Models (dLLMs), on lookahead planning tasks. It identifies a critical asymmetry in planning problems where forward generation requires complex lookahead, but reverse generation is often deterministic, creating an advantage for NAR models. The study demonstrates that NAR models solve planning tasks by decoding backwards from future tokens, requiring exponentially fewer training examples and shallower architectures compared to AR models.
Discrete diffusion models can solve lookahead planning tasks more efficiently than autoregressive models by exploiting the asymmetry between forward and reverse generation.
While Autoregressive (AR) Transformer-based Generative Language Models are frequently employed for lookahead tasks, recent research suggests a potential discrepancy in their ability to perform planning tasks that require multi-step lookahead. In this work, we investigate the distinct emergent mechanisms that arise when training AR versus Non-Autoregressive (NAR) models, such as Discrete Diffusion Models (dLLMs), on lookahead tasks. By requiring the models to plan ahead to reach the correct conclusion, we analyze how these two paradigms fundamentally differ in their approach to the problem. We identify a critical asymmetry in planning problems: while forward generation requires complex lookahead at branching junctions, reverse generation is often deterministic. This asymmetry creates an opportunity for NAR models. Through mechanistic analysis of training and inference dynamics, we demonstrate that NAR models learn to solve planning tasks by utilizing future tokens to decode backwards, avoiding the need to learn complex traversal mechanisms entirely. Consequently, we report that both AR and NAR models are able to achieve perfect accuracy on the lookahead task. However, NAR models require exponentially fewer training examples and shallower architectures compared to AR models, which often fail to converge without specific curriculum adjustments.
