Search papers, labs, and topics across Lattice.
The paper introduces In-Context Reward Adaptation, a transformer-based framework that leverages in-context learning to model diverse and unseen human preferences for RLHF without retraining. They find that standard transformers exhibit an asymptotic bias that hinders adaptation, but that incorporating human response time as an auxiliary input signal enables successful adaptation to unseen preference domains. Experiments demonstrate the framework's ability to represent heterogeneous rewards and handle preference distribution shift, improving the robustness of preference modeling.
Human response time, often discarded, unlocks in-context adaptation to unseen preference domains for RLHF, outperforming standard transformers.
Reinforcement Learning from Human Feedback (RLHF) typically relies on static reward models to align Large Language Models with human preferences. However, human values are inherently diverse and heterogeneous, and a single reward model often lacks the robustness required to generalize to unseen preference domains. While existing multi-reward frameworks attempt to address this, they are often restricted to a fixed set of known domains and fail to adapt to unseen human distributions without costly retraining. In this work, we propose In-Context Reward Adaptation, a transformer-based framework designed to model diverse and unseen human preferences on the fly. By leveraging the in-context learning capabilities of transformers, our approach adaptively infers the underlying reward structure from a small set of preference demonstrations. We demonstrate that while a standard transformer architecture is insufficient for this task by characterizing an asymptotic bias to the ground-truth, incorporating human response time as an auxiliary input signal enables the model to successfully adapt to preferences from previously unseen domains. Our findings show that this approach provides a more robust foundation for preference modeling, allowing for the representation of heterogeneous rewards and preference distribution shift, and offering a scalable path toward more flexible human-AI alignment.
