Search papers, labs, and topics across Lattice.
This paper introduces PVCap, a novel approach to 3D dense captioning that addresses limitations in existing methods by employing a random mixing technique for data augmentation and a robust voxel-based network architecture. By generating pseudo frames with diverse spatial layouts and utilizing a teacher-student framework for pseudo caption labeling, PVCap significantly enhances the model's ability to describe spatial relations in 3D scenes. Experimental results demonstrate that PVCap outperforms the state-of-the-art methods by 11.41% and 13.99% on the ScanRefer and Nr3D benchmarks, respectively.
PVCap achieves a remarkable 11.41% improvement over the state-of-the-art in 3D dense captioning by leveraging innovative data augmentation and a voxel-based architecture.
3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene. Despite the impressive results achieved by previous methods, they suffer from two limitations. First, current research often employs global rigid transformations, such as rotation, to augment scenes without changing their spatial layouts. However, diverse spatial layouts are crucial for training a 3D dense captioning model to describe spatial relations between objects. Second, previous works mainly focus on the design of the caption generation pipeline while utilizing a simple network architecture for other components, i.e., backbone and detection head, which is crucial for extracting rich semantic information for captioning. In this paper, we propose PVCap to alleviate the aforementioned problems. Our PVCap consists of PseudoCap and VoxelCapNet. Specifically, PseudoCap employs a random mixing technique on instances within the dataset, generating numerous pseudo frames with diverse spatial layouts at the instance level. By utilizing a teacher-student framework, PseudoCap obtains pseudo caption labels for these pseudo frames. This data augmentation approach significantly increases the number of training samples and enhances the model's ability to describe the environment effectively. Regarding VoxelCapNet, we introduce a robust caption network that utilizes voxel features and adapts the caption head to the voxel-based network architecture. Our VoxelCapNet can serve as a competitive baseline for future research on 3D dense captioning. Extensive experiments are conducted on two prevalent benchmarks, i.e., ScanRefer and Nr3D. Notably, our method surpasses current state-of-the-art by 11.41% and 13.99% in CIDEr@0.5IoU, respectively. Codes will be made publicly available.
