Search papers, labs, and topics across Lattice.
This paper introduces Token-weighted DPO (TwDPO), a novel training objective for aligning LLMs with human preferences that extends Direct Preference Optimization by weighting tokens based on their importance. The authors instantiate TwDPO with AttentionPO, which leverages the LLM's own attention mechanism to estimate token weights, effectively prompting the LLM to act as a pairwise judge. Experiments demonstrate that AttentionPO outperforms existing preference optimization methods on AlpacaEval, MT-Bench, and ArenaHard, indicating improved alignment and performance.
Attention from your LLM can be used to significantly improve preference optimization, outperforming existing methods without needing a separate reward model or heuristic token weighting.
Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
