Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces ReRef-3D, a benchmark designed to evaluate language-guided placement in 3D scenes, featuring 33,826 instructions across 998 CLEVR-derived scenes. The evaluation method focuses on resolving instructions into valid placement positions while ensuring relation satisfaction and physical validity. After fine-tuning, the models LLaVA-3D, 3D-LLM, and PlaceIt3D achieved valid placement rates of 68.3%, 31.6%, and 22.4%, respectively, highlighting challenges in relation satisfaction, particularly with complex spatial relations.
LLaVA-3D outperforms other models by achieving a 68.3% success rate in valid 3D placements, revealing significant challenges in spatial reasoning.
We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.