Search papers, labs, and topics across Lattice.
The paper addresses the problem of recovering the original layer order of a shuffled 96-layer ResNet given the unlabelled layers and training data. It leverages the negative diagonal structure in the product of input and output projections ($W_{\text{out}} W_{\text{in}}$) of correctly paired residual blocks, derived from stability conditions like dynamic isometry, to identify pairings. The method then uses a hill-climbing algorithm, seeded with proxies like delta-norm or $\|W_{\text{out}}\|_F$, to reconstruct the original layer order by minimizing mean squared error.
Recovering the exact architecture of a trained neural network after randomly shuffling its layers is possible, revealing inherent structural information retained within the weights.
A recent Dwarkesh Patel podcast with John Collison and Elon Musk featured an interesting puzzle from Jane Street: they trained a neural net, shuffled all 96 layers, and asked to put them back in order. Given unlabelled layers of a Residual Network and its training dataset, we recover the exact ordering of the layers. The problem decomposes into pairing each block's input and output projections ($48!$ possibilities) and ordering the reassembled blocks ($48!$ possibilities), for a combined search space of $(48!)^2 \approx 10^{122}$, which is more than the atoms in the observable universe. We show that stability conditions during training like dynamic isometry leave the product $W_{\text{out}} W_{\text{in}}$ for correctly paired layers with a negative diagonal structure, allowing us to use diagonal dominance ratio as a signal for pairing. For ordering, we seed-initialize with a rough proxy such as delta-norm or $\|W_{\text{out}}\|_F$ then hill-climb to zero mean squared error.
