Search papers, labs, and topics across Lattice.
This paper investigates 4-bit key-value (KV) caching to enhance the performance of context-heavy agents, addressing the challenges of long prefix reuse and GPU utilization. By integrating TurboQuant-style rotation and codebook quantization with vLLM FP8 KV caching, the authors achieve significant improvements in task quality and serving throughput. The key result shows that UltraQuant reduces the time-to-first-token by 3.47x in late rounds and increases output throughput by 1.63x compared to the FP8 KV baseline.
UltraQuant slashes time-to-first-token by over 3x in cache-pressured scenarios, revolutionizing how context-heavy agents manage KV caching.
Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV-cache compression for this setting, using TurboQuant-style rotation and codebook quantization as a quality anchor and vLLM FP8 KV caching as the deployment anchor. We report three contributions. First, we frame 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly. Second, we describe the practical design choices needed to make the 4-bit path robust, including asymmetric K/V treatment, Walsh-Hadamard rotation, QJL removal, and block-scale variants. Third, we present serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant, an FP4 approximation path that uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on CDNA4. On a long-context, multi-turn agentic workload, UltraQuant cuts P50 time-to-first-token by 3.47x in the cache-pressured late rounds (2.3x across all rounds) and raises output throughput by 1.63x over the FP8 KV baseline.
