Search papers, labs, and topics across Lattice.
This paper introduces a generative AI-enabled framework for semantic video communication that addresses the challenges of bandwidth constraints and semantic alignment at the network edge. The proposed multi-scale temporal convolutional encoder efficiently captures motion patterns while a capsule-based dynamic routing mechanism refines segment-query associations, allowing for flexible semantic modeling. Experimental results on ActivityNet Captions show significant improvements in recall and mean IoU, achieving 42.9% Recall@0.5 and 41.1% mean IoU, demonstrating the framework's effectiveness for resource-constrained IoT environments.
Achieving over 42% recall in semantic video communication could redefine how we transmit meaning in bandwidth-limited networks.
The exponential growth of video traffic demands novel semantic communication paradigms that transmit meaning rather than raw bits. We present a generative AI-enabled framework for semantic video communication addressing two critical challenges: efficient hierarchical temporal modeling for bandwidth-constrained transmission and robust semantic alignment between video content and natural language queries at network edge devices. Our approach introduces a multi-scale temporal convolutional encoder that captures motion patterns across different temporal granularities with O(T) complexity suitable for resource-constrained IoT deployments. We further propose a capsule-based dynamic routing mechanism that iteratively refines segment-query associations, enabling flexible modeling of non-monotonic semantic alignments essential for goal-oriented communication. These components are unified through a multi-task learning objective optimizing temporal boundary regression, cross-modal alignment, and capsule diversity. Experiments on ActivityNet Captions demonstrate significant improvements, achieving 42.9% Recall@0.5 and 41.1% mean IoU while maintaining computational efficiency critical for edge deployment.
Amazon Science