Search papers, labs, and topics across Lattice.
The authors develop Setwise Relative Policy Optimization (SRPO), a multi-agent reinforcement learning objective that defines the minimal set of outputs driving a single environment transition as a unified action unit. By aggregating member log-ratios into a cardinality-normalized set ratio with a single joint clipping step and relative advantage, SRPO resolves the fundamental mismatch between multi-agent transition dynamics and independent trajectory updates. Across mathematical reasoning and multi-turn search benchmarks spanning four model scales, SRPO establishes state-of-the-art macro-average performance across fixed, mixed, and dynamically routed agent workflows.
Multi-agent LLM training falters when updates are decoupled from joint state transitions; grouping interacting agent outputs into cardinality-normalized set actions solves credit assignment across both static and dynamically routed systems.
Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.
