Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DragOn, a comprehensive benchmark and dataset specifically designed for drag-based GUI interactions, addressing the significant gap in training data for tasks like drag-and-drop and text highlighting. With 286K training screenshots and 3.5M tasks across four domains, DragOn enables a more robust evaluation of both proprietary and open-weight models, including GPT and Claude. The findings indicate that models fine-tuned on this dataset show improved performance on complex drag interactions, highlighting the dataset's potential to enhance the capabilities of GUI agents in real-world applications.
DragOn reveals that fine-tuning on a specialized drag-based dataset can significantly elevate model performance on complex GUI tasks, potentially transforming how we automate digital interactions.
GUI agents - vision-based models that control desktops, web browsers, and mobile devices through graphical user interfaces - promise to automate a wide range of digital tasks. While million-scale datasets have enabled substantial progress on click-grounding, drag grounding (e.g. drag-and-drop, swipe, highlight) data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. We introduce DragOn, a drag grounding benchmark and training dataset covering four domains: text highlighting, cell selection, element resizing and slider manipulation. The dataset comprises 286K training screenshots and 3.5M training tasks, plus a 2000-example held-out evaluation suite. We evaluate proprietary (GPT, Claude) and open-weight (Qwen, Kimi, Holo) models, as well as a Qwen VLM fine-tuned on our training data. Results suggest that our dataset could improve performance of state-of-the-art models on downstream computer-use tasks.