Search papers, labs, and topics across Lattice.
This paper introduces LowRankArena, a standardized evaluation platform designed to facilitate meaningful comparisons of SVD-based low-rank compression methods for large language models (LLMs). By unifying benchmarks, compression budgets, and evaluation protocols, the platform reveals that prior performance claims are highly conditional on specific setups, with significant variations in effectiveness across different model architectures and compression ratios. The findings underscore the necessity for standardized evaluations in the field, as they expose the limitations of nominal low-rank savings and highlight the potential for perplexity degradation that can obscure accuracy improvements.
Prior claims of SVD-based compression effectiveness crumble under standardized evaluation, revealing that performance varies dramatically across models and setups.
SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.
