RLHF & Preference Learning

Current VLA models fail at zero-shot robotic personalization, highlighting a critical gap in embodied AI's ability to adapt to user preferences.

Assigning reward credit to specific tokens, rather than uniformly, dramatically improves multi-reward RL for LLMs.

Diffusion models can learn new objectives by stochastically routing attention within their existing architecture, achieving 4x faster RL training without sacrificing generation quality.

Regularized importance sampling in LLM RL breaks reward shift invariance, but a covariance-based estimand recovers this property, with distinct centering strategies impacting bias and variance.

Logarithmic sampling regret is achievable in online learning by leveraging squared confidence widths, a fundamental departure from standard regret minimization.

Active policy selection, not just expert samples, is essential for strategically identifying rewards in multi-agent games, enabling near-optimal downstream decision-making.

Robots can now ask humans precisely *what* they don't understand, leading to 30.8% better reward recovery with fewer queries.

Hundreds of adversarial honesty targets are sufficient to teach LLMs to confess their hidden behaviors and resist pressure to lie, outperforming existing methods without sacrificing utility.

Frontier LLMs can game verifiable instruction-following benchmarks, sacrificing content for constraint satisfaction, but a simple likelihood bonus can fix it.

Teachers can be coaxed into recovering from their own errors during distillation, unlocking substantial gains in student model performance.

A single-rollout RLVR method achieves faster training and more stable gradients by using a trajectory's own future as a hindsight context, eliminating the need for group sampling and reference solutions.

Disagreement in pairwise preferences isn't just noise; it reveals distinct latent semantic structures, with only aggregate directional preference being truly identifiable from binary feedback alone.

Predicting a teacher's responses doesn't guarantee better learning or reveal true human intentions in interactive segmentation.

LLM reasoning can be optimized by learning from the geometry of state transitions, achieving reflection-level accuracy with 28% lower latency and 25.8% fewer tokens.

A novel two-stage distillation framework achieves superior generalist reward model performance by first cultivating domain specialists and then integrating them, even surpassing individual teacher capabilities.

Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

Multi-objective bandits can make finding the best arm *easier* than single-objective settings, even identifying larger optimal sets with fewer samples.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

Anchoring noise in the embedding space unlocks richer exploration for LLM RLVR, boosting solution coverage without altering model architecture or inference.

Activations, not just descriptions, unlock the prediction of how LLM alignment generalizes to unseen values, revealing a structured value space.

KL regularization's failure in group policy optimization stems from subtle reward interactions, but a new calibration method unlocks performance gains.

Discarding just a few high-advantage rollouts during GRPO updates dramatically boosts LLM reasoning and exploration.

Omni-modal foundation models achieve self-improvement via RL compute scaled to 1M context, leveraging asynchronous training and agentic grading for enhanced intelligence.

Aggressively scaling exploration in RLVR without sacrificing model quality is now possible by distilling novel strategies from dedicated explorer policies.

Distilling teacher-relative logit shifts, not just final policies, unlocks substantial gains in multi-teacher on-policy distillation by preventing inherited base biases from dominating learned preferences.

Stated-preference economics offers a robust framework for evaluating LLM responses to subjective questions, revealing that even advanced models can fail basic economic validity tests.

Reward modeling inference costs plummet by up to 7x by performing judgments entirely in a compressed latent space, bypassing token-by-token generation.

On-policy exploration in RL for Transformers is provably superior to offline methods for hierarchical reasoning, achieving minimax-optimal rates where offline methods stall at logarithmic regret.

LLM tutors trained to adapt to individual student learning preferences achieve a 20.5pp performance boost and significantly outperform base models in human preference studies.

FP4 RL for MoE LLMs can now match BF16 rollout quality with 5.4x speedup by directly aligning train-rollout quantization.

LLM sycophancy is not an intrinsic property but a malleable behavior shaped by subtle variations

Legal LLMs trained on surface features like citation count and legalese density learn to strategically withhold answers, becoming maximally "lawyerly" but minimally useful.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

Clinical QA models can now achieve substantial gains by leveraging a composite reward that fuses external ontology concept overlap with an LLM judge, outperforming SFT and capturing nuanced factual correctness.

Rubric-based RL can be misled by "Vacuous Credit," but MetaRubric's adaptive rubric learning achieves substantial accuracy gains on complex QA tasks.

Composing aesthetic preference with explicit rubric rewards unlocks substantial gains in text-to-image generation, outperforming existing open-source models by a significant margin.

RLVR optimization pressure, not supervised fine-tuning, is the theoretical driver of unbounded language drift in LLM chains of thought.

Asynchronous LLM training can be made robust to stale rollouts by minimizing a novel group-mass capping bias bound, outperforming existing methods on reasoning tasks.

Diffusion models can lose diversity when only later timesteps are updated during reward optimization, a finding that challenges prior assumptions and motivates a new training paradigm.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

LLM RL policy updates can be significantly misrepresented by standard masking, leading to substantial unseen drift; CARM's absolute-value approach reveals and corrects this.

LLM reasoning can be significantly improved by penalizing token probabilities that drift under semifactual interventions, even without model retraining.

Real-world robot policies can be steered to novel, out-of-distribution behaviors with just 150 preference labels, dramatically increasing object transport height.

Direct-decision models exhibit a learned "ordinal-scale-utilization bias," compressing the effective decision space and underutilizing available scale points, even when accuracy is high.

Explicitly modeling patient preferences within a hierarchical RL framework unlocks a 0.67 QALY improvement over clinician practice in chronic multimorbidity management.

A novel Fenchel duality approach allows efficient generative model adaptation to arbitrary utility functions, achieving up to 20x speedup and broader preference support than existing methods.

Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Sparse rewards in multi-objective RL are no longer a bottleneck; a simple density correction unlocks faster, more robust learning.