Search papers, labs, and topics across Lattice.
This paper introduces State Transition Pretraining (STP), a novel approach for scaling GUI agents by optimizing both inverse and forward dynamics on visual state transitions. The method enhances the model's action-grounded visual representations and internal world model, leading to superior performance in desktop and mobile GUI tasks. Empirical results demonstrate that STP-trained models consistently outperform traditional trajectory fine-tuning methods across multiple benchmarks, with performance improvements scaling with the amount of transition data used.
Jointly optimizing action predictions and state transitions leads to significant performance gains in GUI agent training, outperforming conventional methods.
We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.
