Search papers, labs, and topics across Lattice.
The paper introduces Fase3D, an encoder-free 3D Large Multimodal Model that leverages Fourier transforms for efficient point cloud processing. Fase3D addresses the challenges of processing unordered 3D data by using a novel tokenizer that combines point cloud serialization with Fast Fourier Transform (FFT) to approximate self-attention. The model achieves comparable performance to encoder-based 3D LMMs with significantly reduced computational cost and parameter count, demonstrating the feasibility of efficient 3D scene understanding without heavy encoders.
Ditch the heavy 3D encoders: Fase3D uses Fourier transforms to build a surprisingly efficient and scalable 3D large multimodal model directly from point clouds.
Large Multimodal Models (LMMs) that process 3D data typically rely on heavy, pre-trained visual encoders to extract geometric features. While recent 2D LMMs have begun to eliminate such encoders for efficiency and scalability, extending this paradigm to 3D remains challenging due to the unordered and large-scale nature of point clouds. This leaves a critical unanswered question: How can we design an LMM that tokenizes unordered 3D data effectively and efficiently without a cumbersome encoder? We propose Fase3D, the first efficient encoder-free Fourier-based 3D scene LMM. Fase3D tackles the challenges of scalability and permutation invariance with a novel tokenizer that combines point cloud serialization and the Fast Fourier Transform (FFT) to approximate self-attention. This design enables an effective and computationally minimal architecture, built upon three key innovations: First, we represent large scenes compactly via structured superpoints. Second, our space-filling curve serialization followed by an FFT enables efficient global context modeling and graph-based token merging. Lastly, our Fourier-augmented LoRA adapters inject global frequency-aware interactions into the LLMs at a negligible cost. Fase3D achieves performance comparable to encoder-based 3D LMMs while being significantly more efficient in computation and parameters. Project website: https://tev-fbk.github.io/Fase3D.
