Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ByteAction, a foundation model for Byte-space Action Recognition (BAR) that recognizes human actions directly from compressed image bitstreams without pixel decoding. By leveraging a dual-view byte-level recognition framework and innovative techniques like Bitstream Pattern Augmentation (BPA) and Corruption Consistency Training, ByteAction exhibits remarkable robustness against bitstream corruption while preserving competitive performance on intact bitstreams. The model's ability to operate independently of file integrity makes it particularly valuable for privacy-sensitive applications and scenarios involving corrupted data.
ByteAction achieves state-of-the-art robustness against bitstream corruption while maintaining competitive performance on intact data, revolutionizing action recognition in privacy-sensitive environments.
Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.