Search papers, labs, and topics across Lattice.
DRACO addresses the credit assignment bottleneck in outcome-blind, long-horizon agent training by dynamically generating rubrics that track evolving policy capabilities and redistributing trajectory-level scores into step-differentiated advantages for GRPO. By mapping rubric criteria back to responsible actions via a closed-form attribution mechanism, the method circumvents the need for programmatic verifiers or auxiliary learned critic models. The resulting policy outperforms standard GRPO trained with sparse ground-truth rewards by 5.3 points on AppWorld and achieves robust out-of-domain transfer on Tau-Bench.
LLM agents trained without any programmatic verifiers can actually outperform models trained on ground-truth reward signals when trajectory-level rubric judgments are dynamically decomposed into step-level advantages.
Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.
