Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The Image Reconstruction Game introduces a novel benchmark where a vision-language model iteratively instructs an image generator to refine images, allowing for the observation of accumulated common ground. The study reveals that the quality of image reconstruction is predominantly influenced by the describer model, while the generator's capabilities determine the effectiveness of iterative refinements. Notably, the describer's token budget significantly impacts convergence, with shorter budgets leading to more room for improvement but lower initial quality, and longer budgets yielding higher quality at the expense of iterative correction potential.
Describer models dictate image reconstruction quality, revealing that the right balance of token budget can either enhance or hinder the iterative refinement process.
We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image. Benchmarking two Describer models crossed with two Generator models across seven image categories, we find that the describer is the dominant factor in reconstruction quality, while the generator determines whether iterative refinement helps or hurts. Mathematical and geometric images pose the greatest challenge. The describer's token budget strongly affects convergence: shorter budgets yield sparser first renderings with more room for visible improvement, while longer budgets raise absolute quality but leave less to fix. Stronger describers use a richer correction vocabulary spanning spatial, numeric, and structural categories, while weaker describers concentrate on surface properties and tend to stop after a few turns. Human validation shows that the best automated judge reaches only slight-to-fair agreement with human preferences, and automated scores require human recalibration to be used reliably.