Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces TWLA, a post-training quantization framework that achieves significant model compression by employing 1.58-bit weight quantization and 4-bit activation quantization without sacrificing accuracy. The method addresses the challenges of heavy-tailed activation distributions through innovative techniques like the Euclidean-to-Manifold Asymmetric Ternary Quantizer and Kronecker Orthogonal Tri-Modal Shaping. Experimental results confirm that TWLA not only accelerates inference but also maintains high performance across various tasks, making it a viable solution for deploying large language models in resource-constrained environments.
TWLA achieves unprecedented 1.58-bit weight and 4-bit activation quantization while preserving accuracy, revolutionizing the deployment of large language models.
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity. However, existing methods struggle with heavy-tailed activation distributions and therefore keep activations in high precision, fundamentally limiting end-to-end inference acceleration. To overcome this limitation, we propose TWLA, a post-training quantization (PTQ) framework that achieves 1.58-bit weight compression and 4-bit activation quantization while maintaining high accuracy. TWLA comprises three components: (1) Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ) minimizes layer-output error under weight ternarization via a two-stage optimization from Euclidean initialization to manifold relocation; (2) Kronecker Orthogonal Tri-Modal Shaping (KOTMS) applies a Kronecker-structured orthogonal rotation to reshape weights into ternary-friendly tri-modal distributions, while the shared rotation statistically suppresses activation outliers; and (3) Inter-Layer Aware Activation Mixed Precision (ILA-AMP) explicitly introduces adjacent-layer second-order interaction costs in bit allocation and jointly optimizes for the layer-wise disparity of activation quantization gains induced by the shared orthogonal transform, preventing cascades triggered by a few weak layers. Extensive experiments demonstrate that TWLA maintains high accuracy under W1.58A4, while delivering significant inference acceleration. The code is available at.