Search papers, labs, and topics across Lattice.
This paper introduces an unsupervised learning algorithm that employs holographic reduced representations (HRR) to achieve disentanglement in neural networks by treating representations as symbolic structures. By leveraging the HRR unbinding operation, the authors demonstrate a significant improvement in separating factors of variation, outperforming traditional methods like variational autoencoders and generative adversarial networks. The findings are supported by both empirical results and an information-theoretic analysis, revealing that the HRR approach not only enhances robustness to noise but also provides a quantifiable inductive bias for disentanglement.
Holographic reduced representations enable neural networks to achieve superior disentanglement by treating representations as symbolic structures, outperforming traditional methods in robustness and quality.
Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning. Prior work has addressed this problem with variational autoencoders and generative adversarial networks that incorporate ideas from variational inference and information-theoretic constraints. In contrast to methods that rely on continuous representations, we propose a design that treats disentangled representations as symbolic structures, motivated by the compositional relationships among the concepts that make up samples from a distribution. However, learning discrete symbolic structures with neural networks while maintaining differentiability is difficult and often requires complex architectures. To address this, we introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement. We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines, as measured by latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts can be reliably encoded, giving a quantitative account of the inductive bias toward disentanglement. The resulting representations differ from standard autoencoder-based models, in that their latent units are vectors that are summed together, rather than scalar dimensions of a low-dimensional latent vector. We show that this HRR representation is more robust to noise than other disentangled representations and maintains reconstruction quality across a range of SNRs.
