Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of Zero-shot Skeleton Action Recognition (ZSAR) by introducing a novel approach that integrates RGB cues with skeleton data through a classify-by-generation paradigm. By employing a Multimodal Triplet Diffusion model, the authors effectively condition a single class-scoring function on both skeleton and RGB inputs, overcoming the limitations of fixed fusion weights and independent scoring branches. The proposed method achieves superior inductive accuracy on multiple NTU-60/120 splits and outperforms the transductive state-of-the-art on NTU-120, demonstrating the potential of diffusion-based techniques for zero-shot learning tasks.
Classify-by-generation enables superior zero-shot action recognition by dynamically integrating visual and skeletal data without the need for test-time adaptation.
Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context. RGB provides these missing cues, yet existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs. Without using unlabeled test data for adaptation or fusion calibration, a fixed fusion weight cannot capture class-pair-dependent modality reliability, while an adaptive rule lacks target-side feedback for deciding which branch should dominate. We bypass this weight-selection problem via the classify-by-generation paradigm, where each class is scored by how accurately a text-conditioned denoiser predicts the noise added to the skeleton feature. This formulation separates the progressively corrupted skeleton from fixed conditioning, allowing RGB and text to jointly condition a single class-scoring function rather than produce independent scores. We instantiate this idea as Multimodal Triplet Diffusion for Skeleton-Text Matching (TDSM-MM), augmenting a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction. Our proposed TDSM-MM has been ablated via extensive experiments and achieved the best inductive accuracy on three of four NTU-60/120 splits and surpasses the transductive state-of-the-art on NTU-120 96/24 (i.e., 71.3% vs. 69.1%), without test-time adaptation, suggesting that diffusion-based methods can be a promising direction for zero-shot learning.