Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ShadowPEFT is introduced as a novel parameter-efficient fine-tuning (PEFT) method that uses a depth-shared "shadow module" to perform layer-level refinement, evolving a parallel shadow state instead of directly perturbing weights like LoRA. This centralized approach enables parameter reuse across layers, independent pretraining of the shadow module, and detached deployment for edge computing. Experiments demonstrate that ShadowPEFT achieves comparable or superior performance to LoRA and DoRA on generation and understanding tasks, while offering benefits in terms of transferability and inference latency.
Forget distributed weight perturbations: ShadowPEFT's centralized "shadow module" unlocks layer-space adaptation, rivaling LoRA with better transferability and edge deployment potential.
Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.