Search papers, labs, and topics across Lattice.
The paper introduces Post-Optimization Adaptive Rank Allocation (PARA), a data-free post-processing technique to compress LoRA-adapted models. PARA uses Singular Value Decomposition to prune LoRA ranks across layers based on a global singular value threshold, thereby allocating ranks adaptively based on layer importance. Experiments on vision and language tasks show that PARA achieves 75-90% parameter reduction with minimal performance degradation compared to the original LoRA model.
Get 4x-10x smaller LoRA models for free with a simple post-processing step that doesn't hurt performance.
Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90\% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.
