Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces S$^3$, an algebraic framework for specifying and implementing structured sparse patterns in neural networks using Views, Blocks, and Scopes. S$^3$ allows for the precise definition of diverse sparsity structures, including N:M patterns and channel pruning, and supports coupling across tensors for coordinated sparsification. Experiments demonstrate that S$^3$-based implementations of Optimal Brain Damage (OBD) and Surgeon (OBS) outperform existing second-order heuristics in output reconstruction.
Forget hand-coding sparsity: S$^3$ offers an algebraic framework to define and compose structured sparse patterns, unlocking a new level of control and expressiveness in model compression.
We introduce the Structured Sparsity Specification (S$^3$), an algebraic framework for defining, composing, and implementing structured sparse patterns. S$^3$ specifies sparsity through three components: a View that reshapes the tensor via layout composition, a Block specification that defines the atomic pruning unit, and the sparsity decision Scope. Both Block and Scope support Coupling across tensors for coordinated sparsification. S$^3$ enables precise specification of diverse sparsity structures, from fine-grained N:M patterns to coarse channel pruning, and integrates seamlessly with Optimal Brain Damage (OBD) and Surgeon (OBS). We formalize the framework mathematically, demonstrate its expressiveness on canonical patterns, and validate it experimentally via structured OBS and OBD implementations built entirely on S$^3$, which surpasses well-established second order heuristics on output reconstruction across common configurations.