Search papers, labs, and topics across Lattice.
This work introduces MobileSAM2, a lightweight version of the SAM2 model designed for segmenting images and videos on resource-constrained devices. By employing a novel approach called Hypergraphical Knowledge Distill (HyperKD), which utilizes hypergraphs to capture and transfer both temporal and multi-granularity knowledge from the original SAM2, the authors successfully create a model that maintains high performance while significantly reducing resource demands. Extensive evaluations demonstrate that MobileSAM2 achieves promising generalization across various benchmarks, making it suitable for applications in embodied AI tasks on mobile platforms.
MobileSAM2 achieves high-performance segmentation on mobile devices by distilling a heavyweight model into a lightweight version using innovative hypergraph techniques.
The recent large video foundation model, SAM2, enables segment anything in both images and videos, serving as a powerful base model for various applications. However, many of such use cases require to operate on resource-constrained devices like mobile phones and laptops. In this work, we aim to make SAM2 more mobile-friendly by distilling the heavyweight SAM2 into a lightweight model, facilitating segment anything in both images and videos on mobile devices. To this end, we propose Hypergraphical Knowledge Distill (HyperKD), which introduces the idea of hypergraph into knowledge distillation, aiming to effectively model and transfer SAM2's generalizable and comprehensive knowledge. HyperKD consists of Temporal HyperKD and Granularity HyperKD that construct hypergraphs to explicitly model and extract the generalizable temporal knowledge and the comprehensive multi-granularity knowledge from SAM2 respectively, which are then distilled into the lightweight student model by aligning it with the constructed hypergraphs. Besides, we present MobileSAM2, a new family of lightweight SAM2 that balances efficiency and effectiveness via searching the best model architectures with HyperKD during model size reduction. Extensive experiments validate MobileSAM2 across multiple benchmarks and show promising generalization performance on embodied AI tasks.
