Search papers, labs, and topics across Lattice.
This paper addresses the sample inefficiency in reinforcement learning for generative recommenders caused by standard temporal credit assignment, which dilutes the signal for distant but crucial actions. The authors introduce Explanatory World Models (EWM), an attribution framework leveraging a frontier language model's explanations to identify causally relevant past events for reward redistribution. EWM, instantiated with variants inspired by actual causation, explanatory simplicity, and hard-to-vary explanations, significantly outperforms existing methods on long-horizon RL benchmarks and generative recommender tasks, particularly when rewards are delayed.
Language model explanations unlock structured credit assignment, dramatically improving RL sample efficiency in long-horizon generative tasks.
Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
