Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study addresses the modality gap in zero-shot learning for human activity recognition using inertial measurement unit data by evaluating various training and inference configurations on the PAMAP2 dataset. The authors discover that the modality gap is primarily a training-time issue influenced by the encoder objective, with significant improvements in alignment achieved through contrastive training and optimized text prototypes. The best-performing model, which combines contrastive training with inverted softmax correction, achieves a notable accuracy of 73.2% on unseen classes, highlighting the importance of using rich, discriminative descriptions for prototype targets.
Contrastive training can bridge the modality gap in zero-shot human activity recognition, boosting accuracy on unseen classes by nearly 15%.
Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations. We systematically evaluate seven configurations combining three inference methods with two training pipelines on the PAMAP2 dataset, using 14 seen and 4 unseen activity classes with subjects 108 and 109 held out for testing. We find that the modality gap is a training-time phenomenon governed by the encoder objective. A temporal convolutional network (TCN) trained with cross-entropy over label-name Sentence- BERT prototypes yields sensor embeddings with a mean cosine similarity of 0.30 to the corresponding text prototypes, while replacing the label-name prototype targets with discriminative activity descriptions raises this to 0.69. This alignment improvement transfers consistently across all three inference methods. The strongest result combines contrastive training with inverted softmax correction, achieving 73.2% accuracy and 0.583 macro F1 on unseen classes, compared to 58.3% accuracy and 0.34 macro F1 for the label-name baseline. A secondary finding is that richer text descriptions reduce inter-prototype separability in Sentence-BERT space, because shared biomechanical vocabulary causes the language model to compress the prototype cloud. This effect does not negate the benefits of contrastive alignment provided prototype descriptions retain sufficient discriminative vocabulary. We also demonstrate that overall accuracy is a misleading primary metric when test-set class distributions are imbalanced, and recommend macro-averaged F1 as the standard reporting metric for ZSL-HAR benchmarks.