RLHF & Preference Learning - Weekly Roundup

Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Stated-preference economics offers a robust framework for evaluating LLM responses to subjective questions, revealing that even advanced models can fail basic economic validity tests.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

On-policy exploration in RL for Transformers is provably superior to offline methods for hierarchical reasoning, achieving minimax-optimal rates where offline methods stall at logarithmic regret.

Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Stated-preference economics offers a robust framework for evaluating LLM responses to subjective questions, revealing that even advanced models can fail basic economic validity tests.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

On-policy exploration in RL for Transformers is provably superior to offline methods for hierarchical reasoning, achieving minimax-optimal rates where offline methods stall at logarithmic regret.

LLM tutors trained to adapt to individual student learning preferences achieve a 20.5pp performance boost and significantly outperform base models in human preference studies.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

LLM sycophancy is not an intrinsic property but a malleable behavior shaped by subtle variations

Legal LLMs trained on surface features like citation count and legalese density learn to strategically withhold answers, becoming maximally "lawyerly" but minimally useful.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

Clinical QA models can now achieve substantial gains by leveraging a composite reward that fuses external ontology concept overlap with an LLM judge, outperforming SFT and capturing nuanced factual correctness.

Rubric-based RL can be misled by "Vacuous Credit," but MetaRubric's adaptive rubric learning achieves substantial accuracy gains on complex QA tasks.

Composing aesthetic preference with explicit rubric rewards unlocks substantial gains in text-to-image generation, outperforming existing open-source models by a significant margin.

RLVR optimization pressure, not supervised fine-tuning, is the theoretical driver of unbounded language drift in LLM chains of thought.

Asynchronous LLM training can be made robust to stale rollouts by minimizing a novel group-mass capping bias bound, outperforming existing methods on reasoning tasks.

Diffusion models can lose diversity when only later timesteps are updated during reward optimization, a finding that challenges prior assumptions and motivates a new training paradigm.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

LLM RL policy updates can be significantly misrepresented by standard masking, leading to substantial unseen drift; CARM's absolute-value approach reveals and corrects this.

LLM reasoning can be significantly improved by penalizing token probabilities that drift under semifactual interventions, even without model retraining.

Real-world robot policies can be steered to novel, out-of-distribution behaviors with just 150 preference labels, dramatically increasing object transport height.

Direct-decision models exhibit a learned "ordinal-scale-utilization bias," compressing the effective decision space and underutilizing available scale points, even when accuracy is high.

Explicitly modeling patient preferences within a hierarchical RL framework unlocks a 0.67 QALY improvement over clinician practice in chronic multimorbidity management.

A novel Fenchel duality approach allows efficient generative model adaptation to arbitrary utility functions, achieving up to 20x speedup and broader preference support than existing methods.

Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Sparse rewards in multi-objective RL are no longer a bottleneck; a simple density correction unlocks faster, more robust learning.

Legal LLMs can now be optimized for nuanced quality beyond correctness, with rubrics that dissect response evaluation into Style, Element, and Chain of reasoning.