Search papers, labs, and topics across Lattice.
The paper introduces AMIGO, a new benchmark for evaluating agentic vision-language models in long-horizon, multi-image grounding tasks. AMIGO challenges models to identify a hidden target image from a gallery by asking a sequence of attribute-focused questions, stressing question selection, constraint tracking, and fine-grained discrimination. Experiments using the "Guess My Preferred Dress" task demonstrate AMIGO's ability to evaluate identification success, interaction quality, and robustness to oracle imperfections.
Current vision-language benchmarks miss the mark: AMIGO reveals how hard it is for agents to ground visual information across multiple images and turns.
Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn correctness. We introduce AMIGO (Agentic Multi-Image Grounding Oracle Benchmark), a long-horizon benchmark for hidden-target identification over galleries of visually similar images. In AMIGO, the oracle privately selects a target image, and the model must recover it by asking a sequence of attribute-focused Yes/No/Unsure questions under a strict protocol that penalizes invalid actions with Skip. This setting stresses (i) question selection under uncertainty, (ii) consistent constraint tracking across turns, and (iii) fine-grained discrimination as evidence accumulates. AMIGO also supports controlled oracle imperfections to probe robustness and verification behavior under inconsistent feedback. We instantiate AMIGO with Guess My Preferred Dress task and report metrics covering both outcomes and interaction quality, including identification success, evidence verification, efficiency, protocol compliance, noise tolerance, and trajectory-level diagnostics.
