Search papers, labs, and topics across Lattice.
This paper introduces the Hilbert Operator for Progressive Encoding (HOPE), a novel mathematical framework that deconstructs learned representations in deep networks by shifting network compression into a Hilbert space of continuous functions. By modeling neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging into a low-rank subspace projection, allowing for unbiased architectural decisions across various layers. The framework's data-free and hyperparameter-free nature is validated through proof-of-concept experiments demonstrating its effectiveness in model compression and fine-tuning.
HOPE reveals a transformative approach to deconstructing neural representations, enabling unbiased architectural decisions and efficient model compression without the need for data or hyperparameters.
Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.
DeepMind