Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Conditional Program (CP), a novel compressed representation for conditional memory lookup tables in LLMs that reconstructs only requested rows. By encoding each row with a short, two-bit code and a local scale, and sharing a reconstruction program across the table, CP significantly reduces memory footprint. Experiments demonstrate CP's superior KL divergence reduction compared to standard 2-bit quantization and substantial storage savings, enabling larger models or workloads on limited GPU memory.
A two-bit code and shared reconstruction program slash LLM conditional memory storage by 75%, enabling 4x larger tables on GPU.
Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compressed representation that reconstructs only the rows an input requests. CP represents each row by a short code and a local scale, with a reconstruction program shared across the table. We instantiate this representation with a Gaussian trellis, norm restoration and independently decodable row records. CP's two-bit code lowers output KL by 41--63\% relative to 2-bit round-to-nearest. Analysis across three trained n-gram memory families shows that Gaussian-like coordinate statistics coexist with learned relationships among rows; at matched storage budgets, the shared trellis outperforms the fitted codebooks and low-rank residual variants we test. For DeepSeek, it reduces table storage from 203GB to 51GB, leaving room for larger workloads when the table is kept on GPU.