Lei Zhang

K [16]. Following DeiT [72], we develop three variants of BinaryAttention, namely -T (tiny), -S (small) and -B (base), by substituting all standard attention modules with BinaryAttention. We follow the experimental settings in DeiT [72], which are detailed in supplementary file. The models are fine-tuned with the self-distillation [34] strategy, where the full-precision counterparts serve as the teacher. We compare with quantization based methods PTQ, ×\times and 1.