Search papers, labs, and topics across Lattice.
This paper introduces CoCurve, a novel method for structured pruning of large language models (LLMs) that jointly considers the dependencies between computational units, such as attention heads and feed-forward channel groups. By employing a second-order Taylor expansion of the KL divergence between a frozen model and its masked version, CoCurve constructs a Fisher matrix that captures both individual unit saliency and co-pruning curvature, allowing for a more effective pruning strategy. The method achieves significant compression without the need for fine-tuning or additional training, demonstrating that it can outperform traditional independent pruning approaches by preserving essential unit interactions.
Jointly pruning dependent units in LLMs can lead to more effective model compression without the need for fine-tuning, challenging conventional independent pruning methods.
Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.
