Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Adaptive Depth Sparse Framework (AdaDSF), which optimizes pre-trained large language models (LLMs) by converting them into depth-sparse models without requiring full retraining. By leveraging the cosine similarity between layer input and output hidden states, AdaDSF intelligently allocates resources by determining layer-wise token retention ratios and employs a lightweight router for token selection. The approach significantly reduces inference FLOPs while maintaining performance levels comparable to dense models, outperforming existing baselines in terms of accuracy degradation under similar sparsity conditions.
Transforming pre-trained LLMs into efficient depth-sparse models can cut inference costs dramatically without sacrificing performance.
Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost. Existing acceleration methods often rely on task-specific fine-tuning or training from scratch, increasing adaptation cost and limiting cross-task usability. We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained LLMs into depth-sparse models without full retraining. Our key insight is that layers contribute unequally to representation transformation, characterized by the cosine similarity between layer input and output hidden states. Based on this, AdaDSF assigns layer-wise token retention ratios from similarity statistics, uses a lightweight router to select informative tokens at each layer, and introduces a feature-preserving alignment objective to match intermediate and final representations between sparse and dense models. On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts. Under comparable sparsity, AdaDSF consistently yields smaller accuracy degradation than strong baselines including MoD, D-LLM, and DLO.