Search papers, labs, and topics across Lattice.
OneCanvas introduces a novel approach to 3D scene understanding by aggregating patch features onto a single equirectangular panoramic canvas, eliminating the need for complex geometry encoders or extensive training resources. This method allows for effective situated reasoning by maintaining a unified spatial coordinate system across all views, enabling the pretrained Vision-Language Model (VLM) to process the representation as a standard image. As a result, OneCanvas achieves state-of-the-art performance on multiple benchmarks while significantly reducing training compute requirements compared to existing methods.
Achieving state-of-the-art accuracy in 3D scene understanding with an order of magnitude less training compute than traditional methods.
Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than the strongest competing methods.
