Search papers, labs, and topics across Lattice.
This paper introduces SEE, a two-stage data synthesis framework designed to enhance the generation of long-horizon interaction trajectories for GUI agents by leveraging an explicit UI transition graph. The framework addresses the limitations of existing methods that over-rely on human demonstrations and fail to capture rare transitions, resulting in a more comprehensive exploration of app interactions. The results show that agents trained on trajectories generated by SEE achieve an average length of 14.8 steps and demonstrate improved task success and generalization capabilities on unseen screens.
Agents trained with SEE-generated trajectories not only navigate complex multi-step procedures but also achieve unprecedented task success rates in real-world applications.
Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.
