Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SCALE, a Semantic- and Confidence-Aware Listwise Energy-based framework for zero-shot skeleton-based action recognition (ZSAR) that avoids explicit skeleton-text alignment. SCALE uses a text-conditioned Conditional Variational Autoencoder (CVAE) where text representations parameterize the latent prior and decoder, enabling likelihood-based evaluation for unseen classes. The method incorporates a semantic- and confidence-aware listwise energy loss and a latent prototype contrast objective to improve semantic organization and class separability.
Skip brittle skeleton-text alignment in zero-shot action recognition: a lightweight CVAE with a novel loss function achieves state-of-the-art results by ranking classes based on likelihood.
Zero-shot skeleton-based action recognition (ZSAR) aims to recognize action classes without any training skeletons from those classes, relying instead on auxiliary semantics from text. Existing approaches frequently depend on explicit skeleton-text alignment, which can be brittle when action names underspecify fine-grained dynamics and when unseen classes are semantically confusable. We propose SCALE, a lightweight and deterministic Semantic- and Confidence-Aware Listwise Energy-based framework that formulates ZSAR as class-conditional energy ranking. SCALE builds a text-conditioned Conditional Variational Autoencoder where frozen text representations parameterize both the latent prior and the decoder, enabling likelihood-based evaluation for unseen classes without generating samples at test time. To separate competing hypotheses, we introduce a semantic- and confidence-aware listwise energy loss that emphasizes semantically similar hard negatives and incorporates posterior uncertainty to adapt decision margins and reweight ambiguous training instances. Additionally, we utilize a latent prototype contrast objective to align posterior means with text-derived latent prototypes, improving semantic organization and class separability without direct feature matching. Experiments on NTU-60 and NTU-120 datasets show that SCALE consistently improves over prior VAE- and alignment-based baselines while remaining competitive with diffusion-based methods.