Search papers, labs, and topics across Lattice.
This paper introduces ScreenAnnotator, an open-source annotation tool designed to enhance the data quality for training vision-language models (VLMs) by integrating spatial, semantic, and structural information into a unified annotation schema. The tool addresses critical limitations of existing annotation methods, such as expressiveness and reusability, by employing an on-policy annotation loop with a Bayesian Annotation Verifier that significantly improves annotation efficiency and accuracy. Results show that fine-tuning a VLM using data annotated with ScreenAnnotator achieves a 76.1% average accuracy, marking a substantial improvement of 35.1 percentage points over previous methods in flowchart scenarios.
Achieving nearly 100% annotation acceptance on flowcharts and a 35.1% accuracy boost for VLMs, ScreenAnnotator redefines data annotation for complex visual reasoning tasks.
Vision-language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual reasoning. Training models for such advanced capabilities demands a new genre of data that seamlessly unifies spatial coordinates, open-vocabulary descriptions, structured attributes, and topological relationships into a singular representation. However, existing data annotation tools fundamentally fail to meet these intricate demands, suffering from three systematic bottlenecks: limited expressiveness, severe annotation-training decoupling, and poor data reusability. To bridge this infrastructure gap, we introduce an open-source annotation tool, ScreenAnnotator. First, we define a unified annotation atom schema that binds spatial, semantic, and structural primitives into a single unit. Second, we implement an on-policy annotation loop embedded with a Bayesian Annotation Verifier (BAV). Finally, we design a template-driven multi-task data synthesis process dynamically transforms static atoms into diverse multi-dimensional reasoning tasks, eliminating redundant re-annotation. The on-policy loop drives the annotation accept rate to nearly 100% on flowcharts and 77% on GUI screenshots, while steadily reducing per-image annotation time as labeled data accumulate. In the flowchart scenario, fine-tuning a VLM yields 76.1% average accuracy, which is a 35.1% point absolute gain. Our code is available at: https://github.com/WnQinm/Annotator.
