RLHF & Preference Learning - Weekly Roundup

Current VLA models fail at zero-shot robotic personalization, highlighting a critical gap in embodied AI's ability to adapt to user preferences.

Assigning reward credit to specific tokens, rather than uniformly, dramatically improves multi-reward RL for LLMs.

Diffusion models can learn new objectives by stochastically routing attention within their existing architecture, achieving 4x faster RL training without sacrificing generation quality.

Regularized importance sampling in LLM RL breaks reward shift invariance, but a covariance-based estimand recovers this property, with distinct centering strategies impacting bias and variance.

Logarithmic sampling regret is achievable in online learning by leveraging squared confidence widths, a fundamental departure from standard regret minimization.

Current VLA models fail at zero-shot robotic personalization, highlighting a critical gap in embodied AI's ability to adapt to user preferences.

Assigning reward credit to specific tokens, rather than uniformly, dramatically improves multi-reward RL for LLMs.

Diffusion models can learn new objectives by stochastically routing attention within their existing architecture, achieving 4x faster RL training without sacrificing generation quality.

Regularized importance sampling in LLM RL breaks reward shift invariance, but a covariance-based estimand recovers this property, with distinct centering strategies impacting bias and variance.

Logarithmic sampling regret is achievable in online learning by leveraging squared confidence widths, a fundamental departure from standard regret minimization.

Active policy selection, not just expert samples, is essential for strategically identifying rewards in multi-agent games, enabling near-optimal downstream decision-making.

Robots can now ask humans precisely *what* they don't understand, leading to 30.8% better reward recovery with fewer queries.

Hundreds of adversarial honesty targets are sufficient to teach LLMs to confess their hidden behaviors and resist pressure to lie, outperforming existing methods without sacrificing utility.

Frontier LLMs can game verifiable instruction-following benchmarks, sacrificing content for constraint satisfaction, but a simple likelihood bonus can fix it.

Teachers can be coaxed into recovering from their own errors during distillation, unlocking substantial gains in student model performance.

A single-rollout RLVR method achieves faster training and more stable gradients by using a trajectory's own future as a hindsight context, eliminating the need for group sampling and reference solutions.

Disagreement in pairwise preferences isn't just noise; it reveals distinct latent semantic structures, with only aggregate directional preference being truly identifiable from binary feedback alone.

Predicting a teacher's responses doesn't guarantee better learning or reveal true human intentions in interactive segmentation.

LLM reasoning can be optimized by learning from the geometry of state transitions, achieving reflection-level accuracy with 28% lower latency and 25.8% fewer tokens.

A novel two-stage distillation framework achieves superior generalist reward model performance by first cultivating domain specialists and then integrating them, even surpassing individual teacher capabilities.

Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

Multi-objective bandits can make finding the best arm *easier* than single-objective settings, even identifying larger optimal sets with fewer samples.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

Anchoring noise in the embedding space unlocks richer exploration for LLM RLVR, boosting solution coverage without altering model architecture or inference.