Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
OrBIT learns a data-driven, local coding geometry from embedding orbit dynamics to guide the compression of large embedding tables, unlike prior methods that fix geometric constraints. This structure-guided approach utilizes reusable local geometries and a global reconstruction residual to efficiently allocate a fixed coding budget, enabling significant compression with competitive rate-distortion performance. OrBIT achieves over $37.9\times$ compression on GPT-2 and over $23\times$ on 7B models, outperforming established quantization and low-rank baselines.
Embedding compression can be dramatically improved by learning the optimal coding geometry from data, not just optimizing within a fixed one.
Embedding tables are among the largest components of modern language models. Most compression methods fix a coding geometry such as coordinate blocks, low-rank subspaces, or unrestricted codebooks, and optimize within it. We instead ask whether the coding geometry can itself be discovered. We introduce \emph{OrBIT}, a structure-guided embedding compression framework that learns reusable local geometry from orbit dynamics and uses it to constrain a small set of shared codewords. The global reconstruction residual then decides where the fixed coding budget is spent, while redundant overlapping charts let local errors compensate one another after gluing. Our theory shows how tight-chart geometry controls distortion, how the global residual directs sequential allocation, and how data-geometry-guided refinement improves the codec. The resulting orbit machinery is compiled away, leaving a compact decoder in which the learned structure governs what is stored, where capacity is allocated, and how local information is assembled globally. Across four LLM embedding tables, OrBIT achieves $37.9\times$ compression on GPT-2 and over $23\times$ on each 7B table relative to 16-bit storage, while delivering competitive rate-distortion performance against established quantization and low-rank baselines.