Search papers, labs, and topics across Lattice.
This paper introduces a novel task of compositional layout understanding, addressing the limitations of current vision-language models (VLMs) in interpreting complex, hierarchical layouts. The authors present CoDeLayout, a dataset of approximately 20,000 annotated multi-layer layouts, and propose MASON, a post-training paradigm that enhances element interpretation and hierarchical understanding by integrating multimodal alignment and structural perception. Experimental results demonstrate that MASON significantly improves accuracy from 79.68% to 91.66% on the task, while also achieving better performance with less training data compared to traditional fine-tuning methods.
MASON boosts VLM accuracy on compositional layout understanding by over 12% while using only 30% of the training data compared to standard methods.
Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures. In this paper, we introduce a new task, compositional layout understanding, and present CoDeLayout, a VQA dataset of ~20K real-world multi-layer layouts annotated with compositional element pairs and design intent. Through empirical analysis on CoDeLayout, we identify two key challenges for existing VLMs: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter-element relationships. To address these challenges, we propose MASON, a post-training paradigm that integrates multimodal alignment (MA) and structural perception (SP). MA enhances element interpretation by grounding metadata-defined elements to their visual counterparts, mitigating semantic drift, while SP models layer-aware inter-element spatial relationships to improve hierarchical understanding and reduce structural ambiguity. Experiments reveal substantial gaps in existing VLMs: even the strongest baseline, GPT-o3, achieves only 79.68% accuracy, whereas Qwen2.5-VL 7B with MASON reaches 91.66%. Notably, MASON surpasses full-data Direct Finetune using only 30% of the training data and scales better with additional data.
