Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
RoTA-Bench is introduced to evaluate vision-language models' 3D spatial reasoning by requiring them to align 2D projections of rotated objects with target silhouettes within a limited action budget. Empirically, current models exhibit a significant performance gap compared to humans, struggling with disentangling image-space and pose-space progress and demonstrating a reliance on fine-grained interaction history for effective control. These results underscore the difficulty of translating visual observations into precise 3D manipulation capabilities.
Even advanced LLMs fail 83% of the time on a novel 3D spatial reasoning benchmark, revealing a critical gap between 2D perception and precise 3D action control.
Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.