Search papers, labs, and topics across Lattice.
This paper introduces Variance-Calibrated Modulation (VCM), a pre-decoding intervention designed to address the "likelihood trap" in large language models (LLMs) that leads to repetitive and dull outputs. VCM employs two mechanisms: Contextual Searchlight via PMI to enhance contextually relevant tokens while suppressing stopwords, and Adaptive Self-Debiasing to apply scale-invariant penalties based on logit variance. The results demonstrate that VCM significantly improves the diversity and coherence of generated text, especially in open-ended generation and reasoning tasks, without incurring additional computational costs.
VCM reshapes LLM output distributions to enhance diversity and coherence, effectively breaking the cycle of repetitive degeneration in text generation.
In open-ended generation, LLMs frequently fall into the "likelihood trap", marked by repetitive degeneration and vocabulary dullness, creating a discrepancy between machine-generated and human-written text. While post-hoc tail truncation (e.g., Top-$p$, Min-$p$) avoids sampling from the unreliable tail, it can over-sample from the uncalibrated head and misalign generation with human lexical preferences; fixed scalar repetition penalties likewise ignore variation in logit scale across inference steps, potentially disrupting semantic coherence. To address both limitations, we propose Variance-Calibrated Modulation (VCM), a training-free pre-decoding intervention that reshapes the probability distribution before truncation through two dynamic mechanisms: (1) Contextual Searchlight via PMI, which suppresses global stopwords while elevating context-evoked tokens, and (2) Adaptive Self-Debiasing, which uses real-time logit standard deviation for scale-invariant penalization. Across open-ended generation, factual QA, and mathematical reasoning, VCM consistently mitigates the likelihood trap. With negligible computational overhead, VCM integrates with existing decoding strategies, improving diversity, coherence, and, particularly at higher decoding temperatures, reasoning accuracy.
