Search papers, labs, and topics across Lattice.
This paper introduces a weakly-supervised pretraining strategy for skeleton-based zero-shot spatio-temporal action localization, addressing the challenge of high annotation costs by leveraging large-scale action scenery datasets. The proposed Skeleton-Language feature Pooling Switching method transitions from video-level feature aggregation during pretraining to instance-level feature computation during inference without requiring additional training on target actions. Experimental results across four public datasets show significant improvements in estimating unseen actions, highlighting the method's effectiveness in overcoming annotation limitations.
Achieving zero-shot action localization with minimal annotation, this method effectively estimates unseen actions by innovatively leveraging weakly-supervised pretraining.
We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery datasets. Specifically, our approach, termed Skeleton-Language feature Pooling Switching, introduces a weakly-supervised vision-language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the video level and aligns them with each video's known action text embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification datasets demonstrate that the proposed method effectively addresses annotation limitations.
