Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of token reduction and shuffling in Vision Transformers (ViTs) against feature inversion attacks, revealing that these techniques do not adequately protect against privacy breaches. The authors introduce the Spatially Aligned Reconstruction Attack (SARA), which successfully reconstructs input images from transmitted embeddings by predicting token positions and restoring their spatial layout. Their findings indicate that while token reduction provides better protection than shuffling, significant information leakage remains, prompting the introduction of a lightweight edge-side defense that effectively mitigates these vulnerabilities.
Token shuffling may seem to enhance privacy in ViTs, but it fails to prevent substantial information leakage during feature inversion attacks.
Vision Transformers (ViTs) are increasingly used in split-inference systems, where edge devices transmit intermediate token representations to a remote cloud. In this setting, token reduction lowers computation and communication costs, while token shuffling disrupts the spatial organization of the transmitted tokens, potentially limiting information leakage. However, their privacy benefits remain unclear against feature inversion attacks, which attempt to reconstruct the input from the transmitted embeddings. In this work, we show that, despite disrupting the spatial structure required by conventional reconstruction attacks, transmitted token embeddings retain substantial positional information. Based on this observation, we introduce the Spatially Aligned Reconstruction Attack (SARA), a unified pipeline that predicts token positions, restores their spatial layout, reconstructs missing embeddings using a feature-space masked autoencoder, and recovers the input image. Our results demonstrate that token shuffling provides only apparent privacy, as SARA largely reconstructs the original token organization. Token reduction offers stronger protection, but significant leakage persists when the retained tokens preserve sufficient semantic and positional information. Finally, we introduce a lightweight edge-side defense that removes positional embeddings and progressively adapts the edge-side transformer blocks through knowledge distillation. It substantially reduces attack performance against SARA, while preserving downstream task accuracy and requiring no changes to the cloud-side model.