Search papers, labs, and topics across Lattice.
The paper introduces ACTIVEULTRAFEEDBACK, an active learning pipeline for efficient preference data generation in RLHF, focusing on reducing annotation costs. It proposes and evaluates DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods that prioritize response pairs with large predicted quality gaps for annotation. Experiments show that ACTIVEULTRAFEEDBACK achieves comparable or superior downstream performance with significantly less annotated data (as little as one-sixth) compared to static baselines.
Get 6x the RLHF alignment for your LLM with a new active learning pipeline that focuses on annotating the most informative response pairs.
Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.
ETH