Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing post-training quantization (PTQ) methods for classifier-free guidance (CFG) diffusion models, which often overlook the paired structure essential for effective inference. The authors introduce Guidance-Aware Mixed Precision (GAMP), a novel approach that calibrates quantization based on guided predictions and prevents the detrimental effects of branch drift. Empirical results confirm that GAMP significantly enhances sample quality while maintaining efficiency, demonstrating its superiority over traditional methods that fail to account for the CFG model's unique architecture.
Quantized CFG models can achieve high fidelity without sacrificing sample quality, thanks to a new method that prevents branch drift.
Deploying classifier-free guidance (CFG) diffusion models under real-world compute budgets requires quantization, yet existing post-training quantization (PTQ) methods treat CFG models as single-branch networks, ignoring the paired conditional/unconditional structure that CFG inference fundamentally relies on. This structural blind spot has two consequences. At the system level, the two-pass CFG execution pattern imposes a latency overhead that parameter-count and bit-operation metrics conceal entirely, and commodity INT8 inference stacks fail to realize the theoretical efficiency gains that BOPs calculations promise. At the algorithmic level, calibrating against the guidance gap alone admits an exact null space: a quantized model can achieve perfect gap-fidelity diagnostics while the unconditional branch drifts arbitrarily, corrupting every guided prediction at inference time. This paper terms this the branch-drift trap, proves its existence analytically, and confirms it empirically through a false-positive result in which the best-calibrated model by standard diagnostics simultaneously produces the worst sample quality. To close the trap, Guidance-Aware Mixed Precision (GAMP) is proposed, which calibrates directly on the guided prediction, derives per-layer activation-bit sensitivity from guided-output degradation, and allocates bits via a greedy knapsack -- provably preventing unconditional branch drift by construction.
