Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates how optimizer choice influences training data leakage from neural network checkpoints, extending prior work linking KKT conditions to data reconstruction. It demonstrates that the weight-specificity of an optimizer's implicit bias dictates its susceptibility to data recovery, with SGD and Muon retaining sufficient information for practical reconstruction. In contrast, Adam, under specific conditions, encodes only discrete weight information, rendering training sets unidentifiable to first order and exhibiting greater resistance to reconstruction attacks.
Adam's implicit bias offers a practical defense against training data reconstruction attacks, unlike SGD and Muon which leave reconstructible fingerprints.
Releasing a neural network checkpoint can leak information about its training data, and existing reconstruction results rely on gradient-descent training. We ask whether the choice of optimizer affects the ability to recover the training data. Extending work connecting Karush--Kuhn--Tucker conditions to training-data reconstruction, we show that the implicit biases of popular optimizers (SGD, Adam, Muon) leave very different fingerprints of the training data. We argue that the key distinction is the \emph{weight-specificity} of these implicit biases, namely how much continuous information about the final weights the optimizer-dependent target of the stationarity conditions encodes. The limiting stationarity conditions for models trained with SGD and Muon retain enough weight-specific information to reconstruct training examples in practice. For Adam in the $\ell_\infty$ implicit-bias regime, the optimizer-dependent part of these conditions encodes only a discrete pattern of maximal coordinates and their signs, and at nondegenerate KKT points the conditions are too few to identify the training set to first order. Experiments on MNIST and CIFAR-10 support these claims: models trained with Muon can yield stronger reconstructions at much lower training cost than models trained with SGD, although this advantage depends on the training regime and architecture, and models trained with Adam appear consistently more resistant to the evaluated attacks.