RLHF & Preference Learning - Weekly Roundup

RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.

Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.

Treating on-policy distillation as value-based RL slashes rollout compute by 75% while actively preserving the generation diversity needed for high-k reasoning.

Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.

Personalized alignment is fundamentally a candidate-matching problem rather than a generator bottleneck: million-parameter MLPs tapping internal LLM embeddings beat monolithic, billion-parameter reward models across the board at 10,000× lower latency.

Static parallelism layouts leave massive post-training efficiency on the table, but dynamically re-slicing distributed model states across GPUs cuts RL step latency by 28% with less than 0.1% transition overhead.

Treating on-policy distillation as value-based RL slashes rollout compute by 75% while actively preserving the generation diversity needed for high-k reasoning.

Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.

Personalized alignment is fundamentally a candidate-matching problem rather than a generator bottleneck: million-parameter MLPs tapping internal LLM embeddings beat monolithic, billion-parameter reward models across the board at 10,000× lower latency.

Contextual bandits don't actually need reward estimation—exploiting the elicitation geometry of shared monotone links lets algorithms target the decision boundary directly to secure logarithmic regret under severe functional misspecification.

Static problem pools cause LLM reasoning post-training to stall as models saturate the dataset, but dynamically synthesizing frontier tasks via regret-guided procedural generation keeps learning signals saturated for sustained gains.

Long-horizon agent RL frequently stalls because fixed rewards cannot track shifting bottlenecks—co-evolving evaluation rubrics with targeted exploration skills directly breaks the repetitive failure loops of current policies.

Standard RLVR observations are mathematically insufficient to isolate or fix verifier exploitation, proving that reasoning models cannot unlearn rewarded errors without external audit signals.

Image models do not need noisy vision-language reward models to master compositional reasoning: training on programmatically verified synthetic scenes yields a 10x boost in spatial accuracy that transfers directly to natural prompts.

Personalizing LLMs with student profiles improves individual score calibration, yet counterintuitively fails to capture aggregate group-level preferences in educational evaluation.

Step-level PRM guidance in discrete diffusion models actually degrades reasoning performance by up to 13 percentage points compared to simple best-of-N sampling, tripped up by noisy early-step rewards that aggressively prune viable solutions.

Adversarial imitation learning can achieve fast $\widetilde{O}(1/\varepsilon)$ sample complexity across both expert demonstrations and online environment interactions—provided you jointly regularize the policy and the reward.

Uniform token-level distillation often punishes valid student reasoning, but dynamically reallocating teacher supervision using verified outcome agreement unlocks immediate gains across math and code tasks without relying on hard trajectory filtering.

Distilling with verbal feedback usually transfers the teacher's arbitrary stylistic biases rather than its genuine corrections—unless you isolate that feedback as formal probabilistic evidence against next-token hypotheses.

Multi-teacher distillation fails not because of bad routing, but because high-variance instruction feedback quietly drowns out precision domains like mathematics during shared student updates.

RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.

A novel kernel function circumvents gradient starvation in off-policy RLVR, enabling learning from previously discarded positive trajectories.

Timbre drift in multi-turn expressive voice agents can be eliminated by replacing implicit context embeddings with explicitly supervised style instructions optimized via rejection sampling and context-aware DPO.

A diagnostic framework that measures point accuracy alongside dispersion retention, the ratio of predicted to human standard deviation, on 10,000 respondent--question pairs from the World Values Survey (WVS) spanning twelve countries and six continents is developed.

An interval-censored pre-recognition transition, an independent as-of reference standard, and a prespecified recognition proxy are defined to make the claim testable.

A progressive four-stage framework that addresses three clinically critical gaps in TCM prescription generation and substantially improves prescription quality over zero-shot baselines is proposed, with a 7B model (Mistral-7B) surpassing zero-shot GPT-5 on all three TCM evaluation metrics.

Under a local anti-concentration condition on the exit-time law, it is shown that strong exit-time approximation transfers to strong approximation of the failure indicator, enabling existing and future techniques whose applicability depends on improved strong approximation.

Qwen-Audio-Agent is presented, a harness that combines full-duplex voice interaction with asynchronous task execution through a foreground-background architecture that supports the complementary use of direct tool calls for immediate operations and backend delegation for multi-step tasks.