Search papers, labs, and topics across Lattice.
This paper introduces MSVS-VAE, a hierarchical set-based variational autoencoder that enhances 3D reconstruction fidelity while maintaining compactness. By employing a novel point-shuffle upsampling technique and replacing global cross-attention with a local aggregation operator, the model effectively increases spatial capacity for detailed geometry modeling. Experimental results show that MSVS-VAE achieves significantly faster decoding and improved compactness compared to existing voxel-based and set-based approaches, marking a substantial advancement in high-fidelity 3D generative modeling.
Achieving 10x faster decoding and 10x higher compactness, MSVS-VAE redefines the standards for high-fidelity 3D reconstruction.
High-fidelity 3D generative modeling increasingly relies on the latent diffusion paradigm, where the reconstruction quality of the underlying 3D VAE becomes a primary bottleneck. Existing approaches largely follow two paradigms: sparse voxel-based representations achieve strong reconstruction quality but incur significant memory and computational overhead, while set-based representations are compact and continuous yet typically lag in fidelity due to latent sparsity and excessive global smoothness. We propose MSVS-VAE, a hierarchical set-based VAE that closes this fidelity gap without sacrificing compactness. Our key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling. To efficiently decode from the densified hierarchy, we replace global cross-attention with AVS-Conv, a geometry-aware local aggregation operator operating within local neighborhoods rather than the exhaustive latent set. We further introduce multi-scale query decoding to fuse coarse-to-fine latent features, where coarse scales provide stable global context, and fine scales refine localized geometry, reducing artifacts from overly local receptive fields. Extensive experiments on Objaverse, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAE consistently outperforms prior set-based and voxel-based VAEs, delivering approximately 10x faster decoding than prior set-based methods and approximately 10x higher compactness than voxel-based baselines.
