Search papers, labs, and topics across Lattice.
TabuLM is a novel language model specifically pre-trained on Kinyarwanda tabular data, addressing the lack of resources for this morphologically rich language. By enhancing KinyaBERT-large with unique embeddings and a learned attention bias, the model employs two innovative pre-training objectives: Masked Cell Recovery and Column Type Prediction. The results show TabuLM achieving a 62.0% exact match on the new TabQA-kin benchmark, significantly outperforming existing models by up to 12.7 points, highlighting the importance of morphology and table structure in low-resource language processing.
TabuLM outperforms multilingual baselines by over 11 points, showcasing the power of morphology-aware embeddings in low-resource language models.
We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedicated tabular representation learning resource. TabuLM extends KinyaBERT-large, a two-tier morphological transformer, with additive row, column, and cell-type embeddings and a learned table-structure attention bias that sharpens same-row and same-column attention. Pre-training uses two new objectives: Masked Cell Recovery (MCR), which masks entire cells and forces reconstruction from row and column context, and Column Type Prediction (CTP), which predicts column semantic types from observed cell values. We pre-train on 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals, and introduce TabQA-kin, the first native Kinyarwanda table question-answering benchmark comprising 526 QA pairs across 31 tables and four question types. TabuLM achieves 62.0% exact match on TabQA-kin, outperforming KinyaBERT-large by 5.7 EM points and all multilingual baselines (mBERT 49.3%, XLM-R 50.0%) by 11.7-12.7 points. Analysis shows that structural table embeddings are most decisive for comparison and lookup questions, while morphological awareness provides complementary gains. Our code, data, and pre-trained checkpoint are publicly available.
