Search papers, labs, and topics across Lattice.
This paper investigates the issue of text degeneration in pruned large language models (LLMs), particularly focusing on the emergence of repetition loops despite stable perplexity and task accuracy. By analyzing the dynamics of decoding, the authors identify two key factors: loop entry risk and loop persistence, which are influenced by the escape mass of plausible alternatives in token sampling. They introduce two novel token-level guidance objectives鈥擣OCUS and RePAIR鈥攖hat effectively mitigate text degeneration, resulting in reduced repetition and enhanced generation quality in various tasks.
Pruned LLMs can still generate coherent text by strategically guiding token selection, reducing repetition loops and enhancing overall quality.
Pruning is a practical approach to compress large language models (LLMs), but it can amplify text degeneration, especially repetition loops, even when perplexity and task accuracy remain largely unchanged. In this work, we present a token-level analysis of this failure mode by viewing decoding as a dynamical process that enters and persists in a small set of recurrent contexts. Our analysis decomposes degeneration into loop entry risk and loop persistence, and shows that persistence is controlled by the escape mass assigned to plausible alternatives within the token sampling set. Motivated by these findings, we propose two token-level guidance objectives for post-pruning fine-tuning. FOCUS reweights distillation toward high-confidence teacher regions to suppress leakage, while RePAIR uses onset-centered positive/negative continuation pairs with a margin loss to promote plausible alternatives and prevent early commitment to repetition loops. Experiments on open-ended continuation and instruction-based generation show that both methods consistently reduce repetition and improve generation quality.
