RLHF & Preference Learning - Weekly Roundup

This work introduces StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation, and achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems.

The Explainability Assistant is introduced, an open-source conversational XAI system that leverages the function-calling capabilities of modern Large Language Models (LLMs) to overcome limitations and achieves 94% intent-parsing accuracy, supports flexible natural language interaction, and adapts to different ML problem types without task-specific fine-tuning.

This work generates synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer.

Within the tested range, concentration sets neither the destination nor the pace of collapse; the pace follows whose text fills the pool; what emerges instead is an invariance.

COBRA-Skills is introduced, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space and remains robust to changes in the agent harness and performs effectively when the target model itself is used for skill generation and refinement.

This work introduces StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation, and achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems.

The Explainability Assistant is introduced, an open-source conversational XAI system that leverages the function-calling capabilities of modern Large Language Models (LLMs) to overcome limitations and achieves 94% intent-parsing accuracy, supports flexible natural language interaction, and adapts to different ML problem types without task-specific fine-tuning.

This work generates synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer.

Within the tested range, concentration sets neither the destination nor the pace of collapse; the pace follows whose text fills the pool; what emerges instead is an invariance.

COBRA-Skills is introduced, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space and remains robust to changes in the agent harness and performs effectively when the target model itself is used for skill generation and refinement.

This work derives the first model of WhatsApp Web's implementation of the Signal protocol and the most detailed model to date of Signal's original protocol, and reveals previously undocumented differences between the original libsignal library and WhatsApp's fork.

A preference optimization framework with Large Audio-Language Model (LALM) feedback for controllable non-verbal vocalization (NVV) generation in continuous autoregressive speech models that enables DPO-style preference learning without explicit sequence likelihoods while preserving direct supervision on preferred realizations is proposed.

Experiments on 4DMatch and 4DLoMatch show that both variants produce more accurate correspondences than the compared methods and improve downstream registration, with larger gains in low-overlap cases.

A multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection is presented.

LOCUS, a method that selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint, is presented, a method that selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint.

This work introduces MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots, which performs better on reasoning than on planning, and the best-performing model remains far below observed human performance.

Asynchronous LLM agents can now train up to 50 updates off-policy without critic collapse by simply decoupling token-level importance correction from long-horizon reward propagation.

Classical hardness measures miss an entire dimension of bandit exploration: best-arm identification complexity is fundamentally dictated by the Shannon entropy of gap distributions across dyadic scales.

Model scale is no longer the sole bottleneck for complex causal analysis: pairing RL with simulator-synthesized interventions allows a 35B parameter model to surpass Claude Opus 5 on real-world diagnostic tasks.

Outcome-based preference optimization collapses multi-path agent reasoning into brittle single-track policies, but balancing odds across divergence trees preserves viable alternative trajectories and significantly improves error recovery.

Contrastive flow matching can upper-bound forward-KL divergence on fixed preference pairs, resolving the training instability of unbounded DPO surrogates without requiring online rollouts.

Rigorous offline confidence intervals for optimal policy values are finally achievable over diverging horizons and non-stationary data, eliminating the classic non-smoothness bottleneck of maximum Bellman operators.

Skipping gradient updates on rounds without prediction errors completely removes the long-standing $\log T$ horizon dependence in online inverse optimization, securing constant $O(d^2)$ regret for forward integer programs without expensive geometric oracles.

Ride-hailing dispatchers do not need brittle reward engineering: training a preference reward model directly on implicit multi-view passenger and driver trajectories outperforms heuristic-tuned hold policies in live, city-scale deployment.

Optimizing against frozen reward models collapses true executed reasoning performance by 90% under GRPO, but feeding back an on-policy stream of just 10% reality-settled labels closes the hacking gap and preserves 6x the reward.

Sampling rollouts via GRPO actively degrades tool-selection training as policies concentrate, whereas exact policy optimization over enumerable combinatorial tool spaces eliminates vanishing gradients and boosts accuracy by up to 14.2 points.

Instead of stuffing raw context with dialogue history or brittle retrieval vectors, treating user personalization as particle-filtered hypothesis tracing over natural language resolves the tension between fleeting intent and long-term preferences.

Standard single-turn RL produces socially myopic agents, but dynamically scheduling rewards from early relationship-building to mid-dialogue task execution drives a 9.2 percentage point gain in multi-turn goal achievement.

An open-model test-time-compute pipeline for natural-language proof generation for hard olympiad mathematics that operates entirely in natural language, with no formal prover, external tools, or internet access is presented.

Across widely adopted grounding benchmarks and navigation benchmarks spanning multiple platforms, LLaDA-UI substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks, establishing block-wise diffusion as a practical generative paradigm for multimodal GUI agents.

This paper investigates the development path of large language model preference alignment techniques, focusing on the training mechanism of Reinforcement Learning from Human Feedback and its three-stage process, including supervised fine-tuning, reward model training, and PPO-based policy optimization.

Frontier-scale agentic RL on 700B+ MoEs is no longer locked inside hyperscaler proprietary stacks, achieving stable 263-second step times across 64 GB300 GPUs via a verified, open-source training infrastructure.

Nearly 40% of GRPO rollouts yield zero gradient due to uninformative all-or-nothing groups, but an offline anchor pass can slash this cold-start compute waste before the target policy generates a single token.

Formulating preference optimization directly at the gradient level eliminates the notorious over-refusal penalty in open-weight LLMs without compromising downstream utility.

Naive linear reward scalarization inevitably triggers destructive gradient tug-of-wars, but decoupling marginal objectives via orthogonal projections and localized constraints stabilizes training across both recommender systems and complex LLM alignment tasks.

Domain gaps baked in during mid-training are virtually immutable: compensatory SFT failed to bridge a single domain performance disparity at a 5% threshold despite lifting overall baseline accuracy.

Treating all deleted nodes uniformly causes catastrophic utility collapse in graph unlearning, but ordering removals by topological difficulty retains 74% performance at 20% mass deletion where prior baselines plummet to 26%.

AgentGrad is proposed, a prompt optimization framework for multi-agent systems based on sequential intervention and semantic textual gradient abstraction that clusters semantically similar gradients to prevent mixing unrelated failure modes, and abstracts each cluster into a generalized gradient that captures the shared corrective pattern.

Multi-agent LLM training falters when updates are decoupled from joint state transitions; grouping interacting agent outputs into cardinality-normalized set actions solves credit assignment across both static and dynamically routed systems.

Structural alignment—not affective warmth or semantic relevance—is the primary conversational feedback mechanism driving grammatical acquisition in sample-constrained language models.

This work introduces ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans, and introduces ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.

Code generation models can now self-improve at test time without ground-truth unit tests by turning behavioral execution agreement on generated input probes into a stable, hack-resistant policy gradient signal.

Increased availability of verifiers often comes at the cost of their credibility, revealing a critical trade-off in robot policy evaluation.

An on-policy expert-correction pipeline is developed, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn, which preserves the model's planning style and combines the gains of harness evolution and model adaptation.

Co-training speculative draft models directly inside 122B, 256K-token distributed RL runs removes the massive rollout bottleneck without causing pipeline stalls or context-parallel memory blowups.

Alignment faking and evaluation gaming are structural inevitabilities of RL rather than training anomalies, because scalar behavioral scoring cannot theoretically distinguish intrinsic norm adoption from conditional compliance under observation.

DPO-tuned LLMs systematically achieve less than a third of requested emotional intensity because preference optimization cannot reward behavioral extremes that base models never sample.