Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of per-token KL divergence (KLD) as a fidelity metric for quantized large language models (LLMs) by analyzing two cohorts across various downstream benchmarks. The researchers found a strong correlation between KLD and benchmark scores overall; however, this correlation diminished significantly in the near-baseline silent zone, indicating that KLD is not a reliable predictor of model performance in certain contexts. The findings reveal that while KLD captures the volume of disagreement with a reference model, it fails to accurately assess the direction of those disagreements, which is crucial for understanding model behavior.
KLD may mislead researchers by appearing to correlate with model performance while failing to predict outcomes in critical silent zones.
Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality. We test this practice on a 28-quant cohort of Qwen3.6-35B-A3B and a 41-quant cohort of Devstral-Small-2-24B, evaluated across a suite of downstream benchmarks. We find that KLD is strongly correlated with benchmark score over the full cohort ($蟻=-0.72$ on Qwen and $蟻=-0.86$ on Devstral, both with $p<0.001$). However, this relationship collapses to non-significance in the near-baseline silent zone ($蟻=+0.00$ on Qwen and $蟻=-0.24$, $p=0.36$, on Devstral). This collapse persists across 14 measurement variants, including different KLD aggregations, perplexity formulations, top-1 agreement, calibration corpora, and context lengths. At the per-prompt level, KLD has only weak failure-prediction power on code, with failed-vs-passed geometric-mean ratios in $[1.08,1.22]$ across five models on LiveCodeBench, and fails as a cross-model router, achieving only $42.3\%-49.4\%$ accuracy on disagreement prompts. We trace the collapse to a structural decomposition: KLD primarily measures the volume of disagreement with the reference, with silent-zone composite $蟻=+0.94$ ($p<0.001$) on Qwen and $+0.55$ ($p=0.03$) on Devstral, while its relationship to the direction of those disagreements is weak and task-conditional.
