Search papers, labs, and topics across Lattice.
This paper introduces Gated Hindsight Distillation (GHD), a novel training approach for mobile GUI agents that incorporates the next screenshot as privileged information to enhance learning from interaction trajectories. By allowing a teacher model to observe future states and re-score the student's actions only when they fail, GHD effectively enables the agent to learn the rationale behind actions that are not immediately evident. The method demonstrates significant improvements in task success rates over traditional approaches on the AndroidWorld and AndroidLab benchmarks using two vision-language models.
Gated Hindsight Distillation allows GUI agents to learn from future observations, drastically improving their reasoning capabilities in complex environments.
GUI agents are commonly trained offline from successful interaction trajectories. Standard training decomposes each trajectory into prefix-action pairs: the agent predicts an action from the current screen and interaction history, while the subsequent observation is discarded. This removes the rationale of why an action is correct: the evidence often appears only on the subsequent screen. For example, to enable Soft Wrap, the agent should click Edit or View, but nothing reveals this until the menu opens. Without such evidence, standard imitation gives the model little chance of ever sampling and thus learning the correct reasoning. To address this issue, we propose Gated Hindsight Distillation (GHD), which uses the next screenshot as privileged information during training. A student predicts from the observable trajectory prefix, while a parameter-sharing teacher additionally observes the next screenshot and re-scores the student's on-policy responses. We apply distillation only when the student fails and the hindsight-conditioned teacher recovers the demonstrated action. GHD improves task success over GRPO on AndroidWorld and AndroidLab across two vision-language models. The code and checkpoints will be made available.
