Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Permutation-Invariant Vector-Quantized Autoencoder (PI-VQ), a novel approach to discrete representation learning for spatially aligned images that removes positional information from latent codes. This constraint encourages the codes to capture global semantic features, enabling direct interpolation between images without relying on learned priors. To mitigate the reduced information capacity, they propose matching quantization, a vector quantization algorithm based on optimal bipartite matching, achieving a 3.5x increase in effective bottleneck capacity.
Stripping positional information from image representations actually unlocks direct semantic interpolation and one-shot image synthesis, challenging the assumption that spatial awareness is crucial for discrete image latents.
Vector quantization approaches (VQ-VAE, VQ-GAN) learn discrete neural representations of images, but these representations are inherently position-dependent: codes are spatially arranged and contextually entangled, requiring autoregressive or diffusion-based priors to model their dependencies at sample time. In this work, we ask whether positional information is necessary for discrete representations of spatially aligned data. We propose the permutation-invariant vector-quantized autoencoder (PI-VQ), in which latent codes are constrained to carry no positional information. We find that this constraint encourages codes to capture global, semantic features, and enables direct interpolation between images without a learned prior. To address the reduced information capacity of permutation-invariant representations, we introduce matching quantization, a vector quantization algorithm based on optimal bipartite matching that increases effective bottleneck capacity by $3.5\times$ relative to naive nearest-neighbour quantization. The compositional structure of the learned codes further enables interpolation-based sampling, allowing synthesis of novel images in a single forward pass. We evaluate PI-VQ on CelebA, CelebA-HQ and FFHQ, obtaining competitive precision, density and coverage metrics for images synthesised with our approach. We discuss the trade-offs inherent to position-free representations, including separability and interpretability of the latent codes, pointing to numerous directions for future work.