Search papers, labs, and topics across Lattice.
Affiliation:
2
0
3
2-bit KV cache quantization for Looped Transformers achieves 80% memory savings and 4x throughput gains with near-BF16 accuracy by quantizing residuals relative to the final loop's KV states.
Pruned LLMs can still generate coherent text by strategically guiding token selection, reducing repetition loops and enhancing overall quality.
