Search papers, labs, and topics across Lattice.
This study systematically evaluates the effects of post-training quantization on Bangla language understanding in three large language model families: Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B. The findings reveal that while quantization can be beneficial for deployment in Bangla, the performance varies significantly across model architectures and quantization methods, with GPT-OSS experiencing a drastic accuracy drop of up to 57.35% on reasoning-heavy tasks under certain quantization formats. Importantly, the results indicate that model choice and quantization strategy are critical factors, suggesting that practitioners must carefully consider these elements when deploying models on constrained devices.
Quantization can enhance Bangla language model deployment, but the architecture and quantization method can dramatically influence performance, with some models losing over half their accuracy.
Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its effects, however, comes from English benchmarks. It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla, and this gap is what we address here. We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) across five Bangla natural language understanding benchmarks (Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN), using zero-shot evaluation through lm-evaluation-harness. To our knowledge this is the first controlled comparison of quantization formats on Bangla NLU. The three families do not respond the same way: GPT-OSS loses up to 57.35% accuracy on reasoning-heavy tasks under GGUF-W8A16, while Qwen and LLaMA hold steady under GPTQ, and in a few cases the quantized version edges out the full-precision one. BoolQ-BN, a comprehension task, stays stable across all three families regardless of format. Taken together, these results suggest quantization can work well for Bangla deployment, but the choice of architecture and quantization method matters more than the bit width alone. We discuss what this means for practitioners choosing a model to run on constrained hardware.
