Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Squeeze-Release method, which combines iterative pruning with an exact structural minimization technique to significantly reduce the size of neural networks while maintaining their performance. By converting masked networks into smaller dense networks and reintroducing small calibrated noise to previously zeroed-out parameters, the authors effectively reclaim wasted capacity and uncover structural redundancies. The results demonstrate that Squeeze-Release can achieve up to 39x compression for fully-connected networks and 14.8x for modern CNNs, with comparable accuracy, and is extendable to transformer architectures.
Squeeze-Release achieves up to 39x model compression while preserving accuracy, turning wasted capacity into trainable parameters.
Unstructured pruning produces sparse weight tensors, but the standard implementation keeps tensor shapes unchanged so the deployed model is no smaller than before pruning. We present an exact structural rewrite, which we call minimization, that converts a masked network into a smaller dense network with the same forward function up to floating-point rounding. The Squeeze-Release cycle iterates pruning and minimization with an intermediate release step that re-enables the exact-zero positions inside the compacted tensors as small calibrated noise, turning otherwise wasted capacity back into trainable parameters. Successive cycles use that capacity to find structural redundancy a single pass cannot reach. We additionally introduce CompensatedLayerNorm, a function-preserving replacement for LayerNorm that extends minimization to channel reduction across LayerNorm-equipped residual streams. Squeeze-Release compresses the deployable network to 39x smaller than the unpruned model on a fully-connected model network and 14.8x smaller on modern CNN (ConvNeXt-Tiny), at comparable accuracy. In addition we prove that the rewrite can be extended to transformer architectures.