Search papers, labs, and topics across Lattice.
Bielik-Q2-Sharp performs a comparative study of six state-of-the-art 2-bit quantization methods on the Polish Bielik-11B language model, using a shared Hessian matrix for calibration on a Polish corpus. The best variant, QuIP# E8P12, achieves performance statistically equivalent to an IQ2_XXS baseline across 22 Polish benchmarks, while QTIP demonstrates the best per-bit efficiency. The study also identifies a dissociation between log-likelihood preservation and autoregressive generation quality for rotation-based quantization methods.
Two-bit quantization can nearly match the performance of larger models on Polish language tasks, but beware: some methods that look good on paper fail catastrophically when generating text.
We present Bielik-Q2-Sharp, the first systematic academic evaluation of extreme 2-bit quantization applied to a Polish large language model. Using Bielik-11B-v2.3-Instruct (11B parameters, Mistral architecture) as our base model, we compare six state-of-the-art post-training quantization methods -- QuIP#, SpinQuant+GPTQ, ButterflyQuant, QTIP, VPTQ, and AQLM -- all calibrated on a Polish-language corpus (CulturaX-PL) with shared Hessian matrices. Our best variant (QuIP# E8P12) achieves 71.92% across 22 Polish benchmarks versus 72.07% for the IQ2_XXS baseline -- within statistical noise, at a modest size premium (3.26 GB vs. ~2.6 GB). On eq_bench, our method scores 47.14 versus 43.53 (+3.6pp), suggesting superior preservation of higher-order reasoning. QTIP achieves the best per-bit efficiency (79.4% MC acc_norm at ~2.4 bpw, 3.27 GB), matching VPTQ's quality at 35% smaller size. We additionally document a MC-generation dissociation phenomenon where rotation-based methods preserve log-likelihood quality but fail catastrophically at autoregressive generation. The entire project was conducted by a single independent researcher on cloud GPUs (vast.ai) within a $285 budget. All models, Hessians, and evaluation logs are publicly available.
