Search papers, labs, and topics across Lattice.
This study introduces RENDEQ, a generator that creates semantically equivalent render-equivalence sets from scientific figures to measure the agreement-accuracy coupling in vision-language models (VLMs). By systematically evaluating three open-weight VLMs, the authors find that agreement among perturbed inputs can serve as a reliable indicator of correctness, outperforming traditional mean token log-probability baselines in two out of three models. Notably, fine-tuning on cross-render consensus leads to a decline in accuracy, revealing that while agreement can certify correctness, it is contingent on the distribution of model errors.
Agreement among perturbed inputs can mislead accuracy assessments, as fine-tuning on consensus can paradoxically reduce performance.
A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.
