Scalable Oversight & Alignment Theory - Weekly Roundup

Experimental data can actively harm decision-making when external, unchangeable agents act on it, necessitating a new design paradigm beyond standard Bayesian methods.

Logarithmic sampling regret is achievable in online learning by leveraging squared confidence widths, a fundamental departure from standard regret minimization.

Active policy selection, not just expert samples, is essential for strategically identifying rewards in multi-agent games, enabling near-optimal downstream decision-making.

Concept interventions can actively worsen model predictions, and existing "benign leakage" criteria fail to prevent this, necessitating new training objectives.

Robots can now ask humans precisely *what* they don't understand, leading to 30.8% better reward recovery with fewer queries.

Experimental data can actively harm decision-making when external, unchangeable agents act on it, necessitating a new design paradigm beyond standard Bayesian methods.

Logarithmic sampling regret is achievable in online learning by leveraging squared confidence widths, a fundamental departure from standard regret minimization.

Active policy selection, not just expert samples, is essential for strategically identifying rewards in multi-agent games, enabling near-optimal downstream decision-making.

Concept interventions can actively worsen model predictions, and existing "benign leakage" criteria fail to prevent this, necessitating new training objectives.

Robots can now ask humans precisely *what* they don't understand, leading to 30.8% better reward recovery with fewer queries.

LLM agents can be reliably steered back to task goals with a learned, dual-path intervention mechanism that generalizes across tasks and executors.

A single adversarial token can bypass most safety filters.

Prioritizing upstream causes over downstream effects in feature attribution is now computationally tractable, even with complex causal dependencies.

Most self-evolving LLM updates are useless, yet current methods inherit them all; new selection strategies drastically cut harmful inheritance and boost reasoning performance.

LLM agents spontaneously form and dissolve fluid, polycentric governance structures, revealing a fundamental tension between economies of scale and accountability in delegated decision-making.

Cost-penalized active feature acquisition is shown to be a direct Lagrangian relaxation, with a computable certificate for greedy suboptimality and a structural guarantee in the near-submodular regime.

In-context learning emerges significantly earlier than specialized prediction, revealing a fundamental trade-off in how LLMs adapt to novel data based on pretraining scale.

Offline risk-sensitive RL is now provably efficient, with new algorithms and tight bounds for dynamic and static optimized certainty equivalents.

Frontier models can be used sparingly for long-horizon tasks by a weak agent that learns to predict when it needs help, then keeps the strong model on a leash.

SAE features are more compositional than random projections, but pairwise interactions are still essential for accurate behavioral steering.

Agentic RL agents can learn to efficiently manage context and subtasks across multiple sessions by estimating the marginal contribution of each session, outperforming naive advantage broadcasting.

Budgeted adaptation under opponent shift can be solved by dual-counterfactual control that explicitly balances immediate adaptation value against future adaptation loss and local strategic risk.

Language models can be trained to proactively avoid costly errors by learning to "fail gracefully" directly from outcome-level rewards, without needing explicit safety annotations or verifiers.

Efficiently tracking a small, switching subset of experts in continuous action spaces is provably harder than theoretically possible, with polynomial-time algorithms facing inherent regret limitations.

Disagreement in pairwise preferences isn't just noise; it reveals distinct latent semantic structures, with only aggregate directional preference being truly identifiable from binary feedback alone.

Predicting a teacher's responses doesn't guarantee better learning or reveal true human intentions in interactive segmentation.

Feldman's memorization lower bound can be arbitrarily inaccurate, but a corrected version using full-sample posterior probabilities offers an exact replacement.

A hypothesis selection algorithm achieves optimal sample complexity and a near-optimal approximation factor while improving the state-of-the-art runtime by reducing the dependency on failure probability from polynomial to logarithmic.

Agents can recursively improve their problem-solving by learning to adapt their own policy and memory management strategies, achieving substantial error reduction in program selection.

Explicitly training LLM agents on runtime semantics via Harness Policy Distillation unlocks a 7.5x increase in autonomous task success and a 2.5x boost in runtime compliance on TRAJECT-Bench.

Conflicting findings on model representation convergence are unified by a lattice structure, revealing that "Platonic" global agreement is only one facet of a broader representational landscape.

Semi-gradient TD learning with neural networks is provably stable and exponentially converges under a simple entropy-induced relaxation condition.

Transformer context for in-context RL can be compressed to the Krylov dimension, revealing a $Θ(γ^m)$ decay in approximation error below this threshold.

Training an agent to learn from an oracle's future actions, not just its initial decision, unlocks significantly deeper and more robust long-horizon reinforcement learning.

Agents trained with Agent-Reflex exhibit a 29.1% improvement on a noise-heavy benchmark, demonstrating robust failure recovery and proactive reflection even with faulty tool feedback.

Unexplained observations, not just supporting evidence, are the key to LLMs discovering truly novel hypotheses.

System One models can achieve direct, low-latency decision-making by treating answers as structured choices rather than serialized text, with Jeva demonstrating robust null detection and permutation invariance.

Explanations from Natural Language Autoencoders can identify their target activation but fail to discriminate it from plausible rivals, suggesting current training paradigms may not yield robust oversight tools.

A learned "Token-to-Go" progress meter enables multi-turn RL agents to effectively learn from sparse rewards, boosting performance by up to 7.2 points without increasing training costs.

Personal agents mistakenly execute unacknowledged proposals as instructions, even with perfect user memory, due to a fundamental lack of "grounding awareness."

SDE Lyapunov estimates now directly translate to finite-time SGD convergence, simplifying analysis for novel optimizers.

Closed neighborhoods in planar graphs can be taught with at most two non-clashing examples, resolving a long-standing conjecture, yet distinguishing between one and two teaching examples is NP-hard.

Token-wise importance sampling in off-policy RLVR is now provably unbiased and practically superior, outperforming GRPO by up to 2.58% on math benchmarks.

LLM agents can learn to predict their own failure from the very first step of a task, achieving a 4.3x higher success rate in their most confident actions.

Multi-objective bandits can make finding the best arm *easier* than single-objective settings, even identifying larger optimal sets with fewer samples.

Few labels per question are sufficient for near-optimal recalibration when information is pooled across related tasks, with identification regret decaying exponentially with label count.

Representation reuse hinges not on source-task performance, but on preserving task-relevant distinctions and ensuring their consistent interpretability for downstream models.

AI agents can now autonomously rewrite their own code and workflows to improve performance, without retraining model weights.

Anchoring noise in the embedding space unlocks richer exploration for LLM RLVR, boosting solution coverage without altering model architecture or inference.

Agents trained on procedurally generated, harder environments significantly outperform those trained on original benchmarks, even with identical compute and optimization.