Search papers, labs, and topics across Lattice.
This paper explores lossless compression of source code using Large Language Models (LLMs) by introducing two novel symbol-ranking variants that limit predictions to the top-$T$ ranks, effectively improving both compression ratios and throughput. The authors conduct a large-scale evaluation across 30 LLMs, demonstrating that their $T$-bounded approach achieves up to 37% improvement in compression ratio and 40% faster throughput compared to previous LLM-based methods. Notably, their method also outperforms general-purpose compressors like zstd and bzip2, achieving up to 82% relative compression gain, particularly highlighting the unique regularities present in source code that LLMs can exploit.
LLM-based compression techniques can achieve up to 82% better compression on source code compared to traditional methods, revealing hidden efficiencies in code representation.
We study the problem of lossless compression of source code, motivated by the storage demands of large-scale software archives, such as Software Heritage (https://www.softwareheritage.org/). General-purpose compressors (e.g., zstd, bzip2) offer a good trade-off between compression ratio and speed, but fail to exploit all special regularities inherent in source code. Recent approaches leverage Large Language Models (LLMs) within Shannon's symbol-ranking framework, relying on a scheme in which the predicted rank can grow arbitrarily. While effective at reducing space, this setting incurs significant throughput degradation, and leaves open the question whether it is necessary to explicitly encode all ranks. In this work, we introduce LLM-based compressors deploying two novel symbol-ranking variants that bound predictions to the top-$T$ ranks ($T=1$ or $63$), with out-of-threshold symbols stored as exceptions and compressed jointly with the rank stream via general-purpose compressors. We conduct the first large-scale evaluation of LLM-based source code compression across 30 LLMs, including general-domain, code-specialized, and quantized models. Our $T$-bounded approach outperforms prior LLM-based compressors both in compression ratio (up to 37% relative improvement) and compression throughput (40% faster). Compared to general-purpose compressors (e.g., zstd, bzip2), we obtain up to 82% relative compression gain but at a lower speed, thus offering a new trade-off point in the compression-speed spectrum. We also show that these gains are stronger on source code than on natural language, suggesting an interesting indication, namely that source code exposes regularities captured by LLMs but missed by general-purpose exact-match-based compressors. We conclude by commenting on open problems that offer theoretical and practical avenues of research.
