Search papers, labs, and topics across Lattice.
This paper addresses the challenge of fair behavior comparison in open-ended real-world interactions, where agents exhibit multiple valid response styles. The authors introduce ARC (Advantage Regularization via Conditioning), a method that employs strategy-conditioned rollout grouping and hybrid rewards to mitigate distortions in reward-model preferences. Empirical results demonstrate that ARC significantly enhances tool-use benchmarks and reduces time-to-first-token in a new interaction paradigm, highlighting the importance of fair behavioral comparison in optimizing agent performance.
Fair behavior comparisons in agent interactions can dramatically improve performance, reducing response times from nearly 5 seconds to just over 1 second.
Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $\tau/\tau^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.
