Search papers, labs, and topics across Lattice.
This paper introduces TIGER, a continuous gradient inversion attack that optimizes token embeddings to minimize their distance to a subspace derived from transformer gradients, addressing the limitations of previous methods that relied on discrete membership tests. By transforming the subspace signal into a differentiable objective, TIGER achieves significantly improved reconstruction quality and runtime for encoder-only models, while also demonstrating enhanced robustness in decoder models, particularly in the context of differential privacy. The results indicate that TIGER enables successful reconstructions even in federated learning environments that employ DP defenses, marking a notable advancement in the field of privacy attacks on machine learning models.
TIGER achieves unprecedented reconstruction quality in federated learning settings, even under differential privacy constraints, by directly optimizing token embeddings instead of relying on brittle token tests.
Federated learning allows multiple clients to jointly train a shared model by sending gradient updates to a central server while keeping raw inputs local. However, prior gradient inversion attacks show that these updates can reveal enough information to reconstruct client inputs. Existing attacks on transformers either optimize dummy inputs to match the true client updates, which is costly and unstable for modern models, or exploit the low rank of attention gradients to identify a subspace containing the true layer embeddings, followed by a discrete membership test for candidate tokens. However, this token test is brittle under numerical noise, i.e., from quantization or Differential Privacy (DP), and scales poorly for encoder models with non-causal attention. We introduce TIGER, a continuous gradient inversion attack that turns this subspace signal into a differentiable objective. Instead of searching over tokens or matching full gradients, TIGER directly optimizes token embeddings to minimize their distance to the subspace. Our experiments demonstrate that on encoder-only models, TIGER substantially improves both reconstruction quality and runtime over existing attacks, while on decoder models, TIGER is more robust than prior subspace-based attacks, enabling the first successful reconstructions in DP-defended federated learning settings.
ETH