Search papers, labs, and topics across Lattice.
This study introduces the FoR-T2I benchmark to evaluate text-to-image (T2I) models' ability to follow spatial instructions under different frames of reference, highlighting a significant gap in model performance. The authors found that T2I models exhibit a mean accuracy of 41.8% lower when interpreting prompts based on an object's orientation compared to those based on camera view, with the best model achieving only 44.3% accuracy on frame-of-reference prompts. Additionally, they propose a VLM-gated rewriting approach that improves average FoR accuracy from 25.0% to 29.2% using visual feedback, indicating potential pathways for enhancing model performance in spatial reasoning tasks.
T2I models struggle significantly with spatial instructions based on object orientation, achieving only 44.3% accuracy on frame-of-reference prompts.
Spatial instruction following has become a crucial requirement for text-to-image (T2I) generation. A common challenge arises when directional expressions are interpreted under different frames of reference. For example, ``the left of''may refer to the viewer's image coordinates or to the intrinsic orientation of an object, leading to different expected layouts. Existing T2I benchmarks reveal important layout failures, yet they rarely isolate whether models can follow a specified frame of reference when it differs from camera view. To mitigate this gap, we introduce FoR-T2I, a benchmark for evaluating this distinction with 1,200 prompt pairs built from controlled spatial layouts. In each pair, the camera-view (Cam) prompt states the target relation in camera view, while the frame-of-reference (FoR) prompt describes the same target placement through an oriented anchor object. Across 22 closed-source and open-source T2I models, mean final accuracy is 41.8\% lower on FoR prompts than on matched Cam prompts; even the best-performing model achieves only 44.3\% FoR accuracy. This suggests that current models struggle more when the same layout is described through an object's orientation rather than directly in image coordinates. We further analyze this gap by relation type and camera view, compare several training-free prompting and feedback-based mitigation strategies, and propose a VLM-gated rewriting approach that selects rewritten prompts using visual feedback, improving average FoR accuracy from 25.0\% to 29.2\% under the same generation budget.
