Search papers, labs, and topics across Lattice.
This paper introduces DiffVC, a non-autoregressive video captioning framework based on a diffusion model to address the limitations of autoregressive methods, such as slow generation speed and cumulative error. DiffVC employs a discriminative conditional diffusion model that generates textual representations conditioned on visual representations, which are then fed into a non-autoregressive language model for caption generation. Experiments on MSVD, MSR-VTT, and VATEX datasets demonstrate that DiffVC outperforms previous non-autoregressive methods and achieves comparable performance to autoregressive methods with faster generation speed, achieving up to 9.9 CIDEr improvement.
Ditch slow, error-prone autoregressive video captioning: diffusion models can now generate captions in parallel, rivaling autoregressive quality with a significant speed boost.
Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on Diffusion model for Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.
