Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for aligning large language models (LLMs) with user preferences at test time, leveraging a method called REAlignment Reward (REAR). By decomposing the reward function into components related to the question and user preferences, the authors enable a scalable and efficient approach to preference alignment without the need for additional training or data curation. Experimental results demonstrate that REAR significantly outperforms existing test-time baselines and generalizes effectively across various tasks, including mathematical and visual domains.
REAR transforms how we achieve user preference alignment in LLMs, enabling scalable realignment without costly retraining.
Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scalable test-time realignment for preference alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.
