Search papers, labs, and topics across Lattice.
This study investigates the role of phase in the hidden layers of image classifiers, revealing that the identity of images is primarily encoded in the phase rather than the magnitude of their Fourier representations. By transplanting phase information between images in various architectures, including PRISM2D, GFNet, and ViT-B/16, the authors demonstrate that predictions are predominantly influenced by the phase donor, while the magnitude contributes little to accuracy. Interestingly, ResNet-50 initially appears to contradict this finding, but further analysis uncovers a significant latent sign code in its late layers, suggesting that different architectures expose phase information in distinct ways, thereby elucidating the texture-shape gap observed between CNNs and attention models.
Image classifiers rely more on phase than magnitude for identity recognition, challenging conventional views on feature importance in neural networks.
Oppenheim and Lim (1981) showed that natural images stay recognizable when reconstructed from their Fourier phase alone, while the magnitude carries little of their identity. We ask whether trained image classifiers reproduce this asymmetry inside their hidden layers, and we test it causally: given two images, we transplant the phase of one onto the magnitude of the other at a chosen layer and record which image the prediction follows. In PRISM2D, GFNet, and ViT-B/16 the prediction follows the phase or sign donor, and deleting all image-specific magnitude barely moves accuracy, so identity rides on phase while image-specific magnitude is largely dispensable to the readout. ResNet-50 at first seems to break the pattern, because transplanting sign after its ReLUs does nothing; a fair intervention before the ReLU reveals a strong latent sign code in the late blocks, and a DC-only control shows the readout consumes a channel-wise spatial average. Controls rule out the trivial case in which magnitude simply stops depending on the image. The architectures therefore share a phase/sign identity code but expose it in different bases, set by rectification and readout geometry, which gives a mechanistic account of the texture--shape gap between CNNs and attention models.
