Search papers, labs, and topics across Lattice.
This paper introduces ViT-FREE, a multi-exit framework for efficient face recognition that leverages early exiting from pretrained Vision Transformers (ViTs) without the need for retraining. By analyzing the progressive evolution of patch embeddings and attention maps across transformer layers, the authors demonstrate that intermediate representations can provide stable and discriminative features for face verification. The empirical results reveal that exiting at layer 10 achieves a 20% speedup with only a 1.5 drop in verification performance, while a lightweight fine-tuning strategy enhances shallow exit performance without compromising deeper exits.
Exiting from intermediate layers of Vision Transformers can yield significant speedups in face recognition with minimal accuracy loss, revolutionizing deployment on resource-constrained devices.
Vision Transformers (ViTs) have gained significant attention in computer vision and shown strong potential for face recognition (FR). However, their high computational cost makes deployment on resource-constrained devices challenging, motivating the need for methods that balance efficiency and accuracy. In this work, we investigate early exiting in pretrained ViTs as a simple yet effective training-free strategy for efficient FR inference. Leveraging the uniform feature dimensionality across transformer encoder blocks, we introduce ViT-FREE, a multi-exit framework that enables face verification directly from intermediate representations without modifying or retraining the backbone model, and thus, reducing inference cost. Empirically, we show that patch embeddings and attention maps evolve progressively across depth, exhibiting high similarity between consecutive ViT blocks and increasing alignment with the final representation. This indicates gradual feature refinement and attention convergence, suggesting that intermediate layers already provide stable and discriminative representations suitable for early exiting. Through extensive experiments on multiple FR benchmarks, we systematically analyze the accuracy-efficiency trade-off across exit depths. Our results demonstrate that later exits achieve a highly favorable balance, with exiting at layer 10 yielding up to a 20% speedup while incurring only a 1.5 drop in verification performance on benchmarks such as IJB-C. Also, we propose ViT-FREE_FT, a lightweight exit-specific fine-tuning strategy that adapts only the projection layers using a small synthetic dataset while keeping the transformer backbone frozen. This approach improves the performance of shallow exits while preserving the efficiency benefits and leaving deeper exits largely unaffected.
