Search papers, labs, and topics across Lattice.
ReRound is a novel post-training quantization method designed to resolve midpoint ambiguity in standard round-to-nearest (RTN) schemes by employing a conditional diffusion model to reconstruct low-bit weights for pretrained LLMs. By integrating a tolerance metric to guide the rounding direction based on proximity to quantization midpoints, ReRound effectively combines diffusion-based reconstructions with traditional RTN, leading to improved quantization accuracy. The method demonstrates significant performance gains over standard RTN and competes well with calibration-dependent methods, particularly for smaller LLMs, while maintaining offline operation without added inference overhead.
ReRound outperforms standard quantization techniques by resolving midpoint ambiguity, achieving superior accuracy in low-bit weight quantization for small LLMs.
ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.
