Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Parallel Language Models (PLMs) that generate mutually consistent tokens in a single forward pass by utilizing latent auxiliary variables, addressing the inference latency bottleneck of traditional autoregressive models. PLMs are trained via a variational bound combining autoregressive and parallel decoding losses, achieving competitive pretraining objectives and significant inference speedups. Empirically, PLMs demonstrate a superior Pareto frontier compared to autoregressive and discrete diffusion baselines, with a 1.3B parameter model matching autoregressive quality on downstream tasks while achieving 2.3-5.7x faster inference.
Near-autoregressive quality is achievable with parallel decoding, unlocking significant inference speedups across model scales.
Autoregressive language models emit one token per forward pass. This causes high inference latency, in particular as models are scaled up for quality. In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables. PLMs are trained using a variational bound that reduces to an autoregressive loss plus a parallel decoding loss. Empirically, PLMs are on a faster and better Pareto frontier than autoregressive and discrete diffusion baselines. Pretrained PLMs achieve near-autoregressive pretraining loss on model scales of 80M to 3B parameters, and speedup follows a power law in model size. When post-trained, our 1.3B model matches or exceeds an identically trained autoregressive model on math, coding, and instruction-following benchmarks, speeding up inference $2.3\text{-}5.7\times$. Together, we show that parallel generation can be trained from a single training objective at autoregressive quality. We will make our code available at https://github.com/mandt-lab/plm.