Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of top-1 argmax concentration as a collapse warning in fine-tuning discrete diffusion language models (DLMs) using Low-Rank Adaptation (LoRA). The authors find that while top-1 concentration signals a potential collapse across various configurations, it fails to predict actual collapses, leading to a precision of zero. Instead, they propose using the maximum LoRA gradient norm as a more reliable indicator, achieving a precision of 0.68 and F1 score of 0.79 in identifying stable configurations, thus providing a more effective diagnostic tool for DLM training stability.
Top-1 argmax concentration fails as a reliable collapse warning, but max gradient norm offers a promising alternative for identifying stable configurations in DLM fine-tuning.
Discrete diffusion language model (DLM) fine-tuning inherits inexpensive diagnostics from denoising-time confidence monitors, but their PEFT-training meaning is untested. We test top-1 argmax concentration as a collapse warning. Across 816 LoRA/PEFT configurations from three DLM families, the warning fires for every configuration while logs record 0/816 actual collapses at the 200 step horizon, giving zero precision. The cause is pre-equilibrium saturation: top-1 concentration is already high before optimization and quickly becomes insensitive to final training stability. We then evaluate max LoRA gradient norm, a parameter-side signal that samples gradient routing rather than token concentration. On a pooled held-out LLaDA-family split, a train-optimized threshold identifies top-decile final-loss configurations with precision 0.68 and F1=0.79, above the all-positive top-1 baseline even at the lower split-bootstrap confidence bound. Autoregressive controls and cross-family threshold failures bound the result to short-horizon DLM-LoRA inspection rather than a universal collapse detector. Workflow: drop top-1 as a PEFT alarm, log max-gradient early in training, and calibrate thresholds per DLM family before routing runs for inspection.
