Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MASER, a framework that leverages modality-adaptive specialist routing to enhance embodied agents' ability to answer spatially relevant questions in 3D environments. By training five modality adapters on a shared vision-language model backbone and employing a neural routing policy, MASER intelligently selects the most suitable modality based on the semantics of the question. The results demonstrate that no single modality is universally optimal, with point-cloud answers being the best choice in 51.5% of cases, and MASER achieving 51.3% oracle agreement, significantly outperforming a Random-Forest baseline.
No single modality reigns supreme in 3D spatial reasoning—MASER’s adaptive routing achieves over 51% accuracy by selecting the best modality based on question semantics.
In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses. Existing Vision-Language models (VLMs) are fine-tuned over a single modality. This completely ignores the question semantics which may favor a different modality than the finetuned modality. To address this, we propose MASER (Modality-Adaptive SpEcialist Routing), a lightweight framework that trains five different modality adapters of a shared VLM backbone and learns a neural routing policy that selects the best adapter based on the question during inference. We encode each question with a frozen sentence transformer and pass the embedding through a small Multi-layer Perceptron (MLP) trained on oracle adapter-accuracy labels. We evaluate our methodology over the Open3D-VQA benchmark and our evaluations show that no single modality is universally optimal -- point-cloud answers are best in 51.5% of cases. MASER routes with 51.3% oracle agreement, outperforming a Random-Forest ablation (43.5%), with only a single adapter call per question.