Search papers, labs, and topics across Lattice.
This paper introduces VibeWorlding, a comprehensive framework for evaluating and training multimodal agents capable of constructing interactive 3D open worlds based on user queries. The authors create VWE-BENCH, a robust benchmark featuring thousands of 3D assets and multimodal queries, and develop VibeWorlding-Gym, a joint multimodal reinforcement learning framework that enhances agent performance through a sandbox environment and a rubric-based verification system. Experimental results reveal that even advanced models like GPT-5.5 struggle with the task, but the proposed VibeWorlder models demonstrate superior performance, highlighting the potential of RL training in overcoming 3D world editing challenges.
Current leading models fail to achieve over 60% success in constructing 3D worlds from user prompts, but VibeWorlder models leverage RL training to outperform them significantly.
Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare how multimodal agents understand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we propose VibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to even surpass closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.
