Search papers, labs, and topics across Lattice.
Standard outcome-supervised post-training causes LLM agents to suffer severe mode collapse, ignoring alternative valid trajectories from the same decision state. To address this, the authors introduce Direct Diversity Optimization (DDO), an offline post-training framework that maximizes successful strategy coverage by harvesting state-aligned branching sets and optimizing a reference-relative target-odds objective. Evaluated on BabyAI, BabaIsAI, and WebShop, DDO consistently outperforms standard preference baselines and decoding-time diversification in both task success and coverage, while substantially increasing resilience to local action perturbations.
Outcome-based preference optimization collapses multi-path agent reasoning into brittle single-track policies, but balancing odds across divergence trees preserves viable alternative trajectories and significantly improves error recovery.
LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.
