Search papers, labs, and topics across Lattice.
The paper introduces GeoPerceive, a benchmark and data generation pipeline for evaluating geometric perception in VLMs using diagram instances paired with domain-specific language (DSL) representations. They propose GeoDPO, a translator-guided reinforcement learning framework, to enhance VLMs' geometric perception by leveraging an NL-to-DSL translator trained on GeoPerceive's synthetic data to provide fine-grained reward signals. Experiments show that GeoDPO significantly outperforms supervised fine-tuning (SFT) on in-domain, out-of-domain, and downstream reasoning tasks, demonstrating improved performance and generalization.
VLMs can get a +39% boost in downstream reasoning by using translator-guided reinforcement learning to improve geometric perception, a far better result than standard supervised fine-tuning.
Vision-language models (VLMs) often struggle with geometric reasoning due to their limited perception of fundamental diagram elements. To tackle this challenge, we introduce GeoPerceive, a benchmark comprising diagram instances paired with domain-specific language (DSL) representations, along with an efficient automatic data generation pipeline. This design enables the isolated evaluation of geometric perception independently from reasoning. To exploit the data provided by GeoPerceive for enhancing the geometric perception capabilities of VLMs, we propose GeoDPO, a translator-guided reinforcement learning (RL) framework. GeoDPO employs an NL-to-DSL translator, which is trained on synthetic pairs generated by the data engine of GeoPerceive, to bridge natural language and DSL. This translator facilitates the computation of fine-grained, DSL-level scores, which serve as reward signals in reinforcement learning. We assess GeoDPO on both in-domain and out-of-domain datasets, spanning tasks in geometric perception as well as downstream reasoning. Experimental results demonstrate that, while supervised fine-tuning (SFT) offers only marginal improvements and may even impair performance in out-of-domain scenarios, GeoDPO achieves substantial gains: $+26.5\%$ on in-domain data, $+8.0\%$ on out-of-domain data, and $+39.0\%$ on downstream reasoning tasks. These findings underscore the superior performance and generalization ability of GeoDPO over SFT. All codes are released at https://github.com/Longin-Yu/GeoPerceive to ensure reproducibility.
Tsinghua AI