Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ExBind, a controlled diagnostic benchmark that evaluates the visual-to-executable correspondence in multimodal coding systems by isolating the mapping between visual references and their executable counterparts. The benchmark comprises a comprehensive suite of 250 broad cases and 240 targeted cases, allowing for precise assessment of models' ability to generate correct executable references without requiring reasoning traces. Results show that while Qwen2.5-VL-3B achieves high candidate validity (98.4%), it falls short on exact accuracy (76.4%), highlighting the challenges in ensuring precise visual-executable mappings in AI systems.
ExBind reveals that even high-validity models can struggle with exact accuracy, exposing critical gaps in visual-to-executable correspondence in AI systems.
Multimodal coding and editing systems must map a visible or semantic referent to the exact executable object that can be edited. A wrong reference may select a valid but incorrect DOM node, SVG element, graph endpoint, hierarchy member, or table cell, while final execution success alone does not reveal the source of the failure. ExBind isolates this visual-to-executable correspondence layer as a controlled diagnostic benchmark between semantic localization and action execution. It samples representation-independent latent binding instances and compiles them into SVG, DOM, canvas, tree, graph, and table cases with deterministic mappings to executable references. Models output only a strict reference; the evaluator maps predictions back to latent structure and scores structural constraints without requiring reasoning traces. The release contains a 250-case broad suite, a disjoint 240-case targeted suite, and 50 paired latent groups. Qwen2.5-VL-3B achieves 98.4% candidate validity but 76.4% exact accuracy, while Qwen3-VL-4B achieves 100.0% validity and 98.8% exact accuracy. In the targeted table suite, all Qwen2.5-VL-3B residual errors are valid correct-row/wrong-column selections. Candidate-order perturbations change case-level outcomes while preserving this error pattern. ExBind is designed for controlled diagnosis rather than population-scale ranking or end-to-end editing evaluation. Code and benchmark records are available at https://github.com/Daerwang2020/Exbind and https://huggingface.co/datasets/Ziqianwwww/ExBind.
Tsinghua AI