Search papers, labs, and topics across Lattice.
To evaluate whether multimodal models understand structured visual compositions beyond flat pixel representations, the authors introduce LayerWiseBench, a programmatic benchmark of 2,800 multi-layer charts assessing layer attribution, semantic binding, and visibility ordering across 7,329 QA probes and 53,791 instruction-guided editing tasks. The evaluation reveals a severe structural blind spot: while top VLMs like Qwen3.5-27B score above 93% on attribution and binding, visibility ordering accuracy plunges to 61.46%, and modern image editors fail catastrophically with overall mIoUs under 5%. These results demonstrate that front-to-back occlusion and component identity remain unsolved bottlenecks for both discriminative and generative multimodal architectures.
Multimodal models master chart semantics with near-perfect accuracy, yet fundamentally collapse when reasoning about 2.5D visual occlusions鈥攄ropping to 61% on layer ordering while generative editors fail under visibility constraints with mIoUs as low as 0.37%.
Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.
