RLHF & Preference Learning - Weekly Roundup

RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.

LLM reasoning can be significantly improved by penalizing token probabilities that drift under semifactual interventions, even without model retraining.

Real-world robot policies can be steered to novel, out-of-distribution behaviors with just 150 preference labels, dramatically increasing object transport height.

Direct-decision models exhibit a learned "ordinal-scale-utilization bias," compressing the effective decision space and underutilizing available scale points, even when accuracy is high.

Explicitly modeling patient preferences within a hierarchical RL framework unlocks a 0.67 QALY improvement over clinician practice in chronic multimorbidity management.

LLM reasoning can be significantly improved by penalizing token probabilities that drift under semifactual interventions, even without model retraining.

Real-world robot policies can be steered to novel, out-of-distribution behaviors with just 150 preference labels, dramatically increasing object transport height.

Direct-decision models exhibit a learned "ordinal-scale-utilization bias," compressing the effective decision space and underutilizing available scale points, even when accuracy is high.

Explicitly modeling patient preferences within a hierarchical RL framework unlocks a 0.67 QALY improvement over clinician practice in chronic multimorbidity management.

A novel Fenchel duality approach allows efficient generative model adaptation to arbitrary utility functions, achieving up to 20x speedup and broader preference support than existing methods.

Distilling expert LLM behavior is now possible without reference solutions by using a fixed expert's next-token predictions on the student's *prefixes* as privileged context.

Smaller, cheaper models generate superior negative examples for preference distillation, significantly boosting student performance without requiring larger teacher models.

Sparse rewards in multi-objective RL are no longer a bottleneck; a simple density correction unlocks faster, more robust learning.

Legal LLMs can now be optimized for nuanced quality beyond correctness, with rubrics that dissect response evaluation into Style, Element, and Chain of reasoning.

Training a small advisor to steer frozen frontier LLMs actually degrades if it learns from every post-hoc correction—selectively distilling only the advice that causally shifts execution yields up to a 6.4% gain and robust cross-family model transfer.

GRPO wastes roughly half its rollouts on zero-variance batches, but reparameterizing pass rates onto the Bernoulli Fisher–Rao arc length turns prompt curriculum design into a closed-form Gaussian selection problem that cuts sampling compute in half.

Heterogeneous LLMs can fix each other's dead-end zero-reward rollouts in RLVR through off-policy trajectory swapping, boosting math reasoning by up to 4.5 points without extra compute or a stronger teacher model.

Driver interventions are not cruise control failures—they provide rich, implicit supervision that overcomes the persistent over-smoothing problem in end-to-end driving models.

Exposing the critic to both successful and failed same-prompt rollouts as contrastive foils resolves PPO's notoriously brittle credit assignment in multi-step reasoning—retaining its efficacy even when paired with a downscaled critic.

On-policy LLM post-training's generalization advantage stems from its dynamic parameter update direction, a principle transferable to SFT for improved performance and efficiency.

Large-scale rewards can unfairly suppress learning signals from smaller rewards in multi-objective RL; CorrGRPO fixes this by normalizing reward covariances.

Outcome-only RL can train highly effective process supervision models by rewarding accurate outcome prediction, even without explicit step-level feedback.

Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.

Treating on-policy distillation as value-based RL slashes rollout compute by 75% while actively preserving the generation diversity needed for high-k reasoning.

Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.

Personalized alignment is fundamentally a candidate-matching problem rather than a generator bottleneck: million-parameter MLPs tapping internal LLM embeddings beat monolithic, billion-parameter reward models across the board at 10,000× lower latency.

Contextual bandits don't actually need reward estimation—exploiting the elicitation geometry of shared monotone links lets algorithms target the decision boundary directly to secure logarithmic regret under severe functional misspecification.

Static problem pools cause LLM reasoning post-training to stall as models saturate the dataset, but dynamically synthesizing frontier tasks via regret-guided procedural generation keeps learning signals saturated for sustained gains.

Long-horizon agent RL frequently stalls because fixed rewards cannot track shifting bottlenecks—co-evolving evaluation rubrics with targeted exploration skills directly breaks the repetitive failure loops of current policies.

Standard RLVR observations are mathematically insufficient to isolate or fix verifier exploitation, proving that reasoning models cannot unlearn rewarded errors without external audit signals.

Image models do not need noisy vision-language reward models to master compositional reasoning: training on programmatically verified synthetic scenes yields a 10x boost in spatial accuracy that transfers directly to natural prompts.

Personalizing LLMs with student profiles improves individual score calibration, yet counterintuitively fails to capture aggregate group-level preferences in educational evaluation.

Step-level PRM guidance in discrete diffusion models actually degrades reasoning performance by up to 13 percentage points compared to simple best-of-N sampling, tripped up by noisy early-step rewards that aggressively prune viable solutions.

Adversarial imitation learning can achieve fast $\widetilde{O}(1/\varepsilon)$ sample complexity across both expert demonstrations and online environment interactions—provided you jointly regularize the policy and the reward.

Uniform token-level distillation often punishes valid student reasoning, but dynamically reallocating teacher supervision using verified outcome agreement unlocks immediate gains across math and code tasks without relying on hard trajectory filtering.

Distilling with verbal feedback usually transfers the teacher's arbitrary stylistic biases rather than its genuine corrections—unless you isolate that feedback as formal probabilistic evidence against next-token hypotheses.

Multi-teacher distillation fails not because of bad routing, but because high-variance instruction feedback quietly drowns out precision domains like mathematics during shared student updates.

RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.

A novel kernel function circumvents gradient starvation in off-policy RLVR, enabling learning from previously discarded positive trajectories.