Search papers, labs, and topics across Lattice.
This study investigates the transparency of the DiffusionGemma model, focusing on variable and algorithmic transparency to understand its reasoning processes. Initial findings indicate that DiffusionGemma exhibits significantly higher opaque serial depth compared to the autoregressive Gemma 4 model, but this can be mitigated by treating intermediate states as interpretable, reducing the opaque depth substantially. The research also identifies unique diffusion-specific phenomena that challenge traditional notions of reasoning in LLMs, while demonstrating that DiffusionGemma maintains comparable monitorability to its predecessor.
DiffusionGemma's reasoning may seem opaque, but by interpreting its intermediate states, we can dramatically enhance transparency without sacrificing performance.
LLM reasoning transparency is a critical affordance for understanding model decisions, mitigating misuse and misalignment, and debugging surprising model behaviors. However, DiffusionGemma performs a larger fraction of its computation in a continuous latent space; does this make its reasoning less transparent? We study this question by decomposing transparency into two components: variable transparency, whether we understand intermediate snapshots of a model's computational state; and algorithmic transparency, whether we can use these snapshots to reconstruct the process by which the model arrived at its outputs. Naively, DiffusionGemma has poor variable transparency: its opaque serial depth, the amount of serial computation that occurs in between interpretable model states, seems at first 28.6X higher than the corresponding autoregressive Gemma 4 model. However, we show that we can map the information flowing between denoising steps through an interpretable token bottleneck with no decrease in downstream performance. Treating these intermediate states as interpretable reduces the opaque serial depth to just 1.1X that of Gemma 4. Algorithmic transparency is harder for diffusion models than for autoregressive models because all token predictions in the canvas can change at every denoising step, giving the model the power to implement complicated distributed algorithms during the denoising process. To begin bridging this gap, we conduct a suite of interpretability case studies, uncovering initial evidence of novel diffusion-specific phenomena such as non-chronological reasoning, token and sequence smearing, and intermediate-context reasoning. Finally, we test monitorability, a key application of transparency that measures whether model outputs are useful for downstream tasks. We find that DiffusionGemma is similarly monitorable to Gemma 4.
