Search papers, labs, and topics across Lattice.
This paper introduces AlphaToken, a novel framework for response token valuation in large language models (LLMs) that separates the objectives of adaptation and stability while incorporating path-awareness in token selection. By leveraging a Fisher-drift proxy to approximate stability and employing an efficient token-level valuation method, AlphaToken enhances fine-tuning by masking low-value tokens and focusing on those that contribute more significantly to performance. Experimental results demonstrate that AlphaToken not only boosts post-training performance but also effectively reduces catastrophic forgetting in LLMs.
AlphaToken reveals a method to enhance LLM post-training by strategically valuing response tokens, leading to improved performance and reduced forgetting.
Token selection is pivotal for effective LLM post-training. However, existing methods mostly rely on local heuristics and rarely formulate token selection as a principled valuation of individual response tokens. We introduce $\textbf{AlphaToken}$, a response token valuation framework that decouples valuation into $\textbf{adaptation}$ (promoting target-task learning) and $\textbf{stability}$ (preserving pre-trained capabilities), and makes each objective $\textbf{path-aware}$ by combining the direct-path signal from local token gradients with the downstream causal-path signal in autoregressive generation. Since retention data are typically unavailable, AlphaToken approximates stability via a $\textbf{Fisher-drift proxy}$ anchored at the pre-trained reference model. For efficient computation, we extend Ghost Dot-Product to token-level valuation. AlphaToken masks low-value response tokens during fine-tuning and preference optimization, concentrating training signals on more valuable positions. Experiments show that AlphaToken improves post-training performance and mitigates catastrophic forgetting.
