Search papers, labs, and topics across Lattice.
The paper investigates using weak LLMs as annotators for preference alignment, finding that high-confidence samples from these models can outperform full human annotations. They introduce Confidence-Weighted Preference Optimization (CW-PO), a framework that re-weights training samples based on a weak LLM's confidence score. Experiments demonstrate that CW-PO, using only 20% of human annotations, surpasses the performance of models trained with 100% human-labeled data under standard DPO.
Weak LLMs, when strategically leveraged via confidence-based sample weighting, can not only drastically cut preference alignment costs but also surpass the performance of models trained on full human-labeled datasets.
Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.
