Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a mathematically rigorous derivation of the reward-to-go policy gradient estimator, clarifying the "causality" argument often used in its presentation. It decomposes the policy gradient objective over prefix trajectories, showing that reward-to-go arises directly from this decomposition rather than as a post-hoc replacement for the full return. The paper demonstrates that the standard causality argument is a corollary of this derivation, offering a more conceptually sound understanding of the estimator.
Reward-to-go isn't just a trick, it's baked into the math from the start: a new derivation reveals it arises directly from decomposing policy gradients over trajectory prefixes.
In introductory presentations of policy gradients, one often derives the REINFORCE estimator using the full trajectory return and then states, by ``causality,'' that the full return may be replaced by the reward-to-go. Although this statement is correct, it is frequently presented at a level of rigor that leaves unclear where the past-reward terms disappear. This short paper isolates that step and gives a mathematically explicit derivation based on prefix trajectory distributions and the score-function identity. The resulting account does not change the estimator. Its contribution is conceptual: instead of presenting reward-to-go as a post hoc unbiased replacement for full return, it shows that reward-to-go arises directly once the objective is decomposed over prefix trajectories. In this formulation, the usual causality argument is recovered as a corollary of the derivation rather than as an additional heuristic principle.