Search papers, labs, and topics across Lattice.
This study explores the potential of using activation-based label propagation to adapt large language models (LLMs) to user-specific preferences in low-resource settings, where traditional human annotation is impractical. By analyzing the distinct clustering of activations from chosen and rejected responses, the authors reveal that while aligned LLMs perform poorly for non-mainstream preferences, a lightweight linear probe trained on a small number of labeled preference pairs can effectively annotate large unlabeled datasets. The proposed method not only outperforms direct training with the same annotation budget but also competes well against models trained on significantly larger datasets, demonstrating its efficiency and effectiveness in preference optimization.
A lightweight linear probe can unlock effective preference adaptation in LLMs with minimal labeled data, outperforming traditional methods.
Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs ($\leq$500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on $50-100\times$ more labelled data in the majority of our settings. Code is available at https://github.com/alessioGalatolo/activ-pref-probe.
