Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This work recasts anchoring in diffusion language models as latent-space caching, decoupling inference into an intermittent, compute-heavy anchor network that tracks persistent global semantics and a lightweight denoising network that runs at every step. By enabling self-supervised time-based anchoring, the method eliminates the requirement for supervised important-token targets while maintaining high semantic fidelity across diffusion steps. Applied to DiffusionGemma-26B, this strategy boosts inference throughput by 49% to 79% across challenging reasoning benchmarks including AIME26, GPQA-Diamond, and HumanEval, while reducing pretraining layer compute by up to 38%.
Diffusion language models can slash generation latency by up to 79% simply by treating latent semantic anchors as an intermittent cache rather than recomputing full network states at every denoising step.
Recent work on anchored diffusion language models improves denoising by shaping an intermediate latent space with supervised important-token targets. In this work, we introduce time-based (self-supervised) anchoring, which learns and reuses latent anchors without requiring such targets. Our key observation is that anchors encode persistent properties of the clean sequence, such as its semantic intent, global structure, or intermediate plan. Although their hidden representations become stale as the token canvas evolves, their semantic content remains useful across nearby diffusion times. This is implemented through a two-stage architecture consisting of a relatively expensive anchor network that generates the latent cache state and a lightweight denoising network that intelligently combines the cached latent state with the current state at each reverse step using a fusion module. This gives anchoring a latent-space caching interpretation: the anchor network is evaluated periodically, while its cached representation is reused across multiple reverse steps. We instantiate this framework as TADM:Post-train, which time-anchorizes pretrained DLMs, and TADM:Pretraining, which learns time-based anchors during pretraining. Applied to DiffusionGemma-26B, TADM:Post-train improves throughput by approximately 49% to 79% on several math, code, and STEM benchmarks (GSM8K, AIME26, GPQA-Diamond, LiveCodeBench-v6, HumanEval, MMLU-Pro). TADM:Pretraining reduces Transformer-layer computation by up to 38% relative to a standard single-stage DLM, achieves up to 73% higher measured throughput than ADLM.