Search papers, labs, and topics across Lattice.
This paper explores knowledge distillation techniques to accelerate 3D hand mesh reconstruction using lightweight neural networks. They focus on boosting the HaMeR model by replacing its ViT-H backbone with lighter alternatives like MobileNet and ConvNeXt, combined with output-level, feature-level, and hybrid knowledge distillation strategies. Results show that lightweight backbones (35% the size) achieve 1.5x faster inference with only a 0.4mm accuracy difference, demonstrating the effectiveness of knowledge distillation for efficient hand reconstruction.
Shrinking a leading 3D hand mesh reconstruction model by 65% yields a 1.5x speedup with minimal accuracy loss, unlocking real-time performance on resource-constrained devices.
Fast and accurate 3D hand reconstruction is essential for real-time applications in VR/AR, human-computer interaction, robotics, and healthcare. Most state-of-the-art methods rely on heavy models, limiting their use on resource-constrained devices like headsets, smartphones, and embedded systems. In this paper, we investigate how the use of lightweight neural networks, combined with Knowledge Distillation, can accelerate complex 3D hand reconstruction models by making them faster and lighter, while maintaining comparable reconstruction accuracy. While our approach is suited for various hand reconstruction frameworks, we focus primarily on boosting the HaMeR model, currently the leading method in terms of reconstruction accuracy. We replace its original ViT-H backbone with lighter alternatives, including MobileNet, MobileViT, ConvNeXt, and ResNet, and evaluate three knowledge distillation strategies: output-level, feature-level, and a hybrid of both. Our experiments show that using lightweight backbones that are only 35% the size of the original achieves 1.5x faster inference speed while preserving similar performance quality with only a minimal accuracy difference of 0.4mm. More specifically, we show how output-level distillation notably improves student performance, while feature-level distillation proves more effective for higher-capacity students. Overall, the findings pave the way for efficient real-world applications on low-power devices. The code and models are publicly available under https://github.com/hunainahmedj/Fast-HaMeR.
