Search papers, labs, and topics across Lattice.
This paper investigates depth pruning in LLMs from a functional perspective, arguing that layer redundancy is jointly determined by the model and the evaluation objective, rather than being an inherent structural property. Through experiments on three LLM families using two calibration objectives and seven search algorithms, the authors find that different objectives lead to qualitatively different redundant layers, with perplexity and downstream accuracy rankings showing inconsistent alignment. The study concludes that the calibration objective is more influential than the search algorithm, suggesting that future research should focus on objective design for depth pruning.
The secret to effectively pruning LLMs might not be *how* you search for redundant layers, but *what* you're optimizing for.
Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work has focused on importance criteria and search algorithms, often treating layer redundancy as an inherent structural property of pretrained networks. In contrast, we adopt a \emph{functional perspective}, where redundancy is jointly influenced by the model and the evaluation objective, suggesting that a universal ranking may not be sufficient. Through an empirical study across three LLM families, two calibration objectives, and seven search algorithms, we observe that different objectives yield qualitatively different redundant layers, and that perplexity and downstream accuracy rankings do not consistently align. Under a fixed objective, however, search algorithms tend to produce similar solutions. Overall, our results suggest that the calibration objective may play a more influential role than the choice of search algorithm, indicating that further attention to objective design could be beneficial.
