Search papers, labs, and topics across Lattice.
This paper introduces Token-Level Off-Policy Labeling (TOPL), an innovative training paradigm that transforms post-training into a token-level correctness prediction task, enhancing model performance in generating accurate responses. By focusing on distinguishing good and bad tokens, TOPL effectively improves out-of-distribution generalization in document summarization across 11 datasets, outperforming various sequence-level and token-level baselines. Additionally, the method demonstrates strong transferability to machine translation tasks, with ablation studies confirming the necessity of token-level learning signals for optimal performance.
Achieving superior out-of-distribution generalization, TOPL reveals that focusing on token-level correctness can significantly enhance model performance across diverse generation tasks.
We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.
