Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors evaluate selective classification via split conformal prediction in multilingual, low-resource settings across African languages on MasakhaNEWS and AfriSenti. They find that standard pooled calibration masks catastrophic group-level failures—meeting an aggregate 90% accuracy target while coverage collapses to 77.5% on specific low-resource or zero-shot languages. Calculating language-conditional deferral thresholds with just 100–200 calibration labels restores 89–91% empirical coverage across all languages without retraining, while exposing stark operational disparities where human review loads swing from under 8% to over 80%.
Standard pooled conformal prediction hides critical safety failures in multilingual models, allowing individual African languages to silently drop to 77.5% reliability despite satisfying a global 90% guarantee.
In the Global South, the lower-income countries of Africa, Asia, and Latin America where most of the world's languages are spoken, a deployed text classifier usually runs on ordinary CPUs, serves many languages with a single model, has few labeled examples in any of them, and relies on people to catch its mistakes. Such a system is only useful if it can promise how often it will be wrong: at most a fixed fraction of the labels it assigns on its own may be incorrect, and everything else must go to a person. Split conformal prediction delivers this promise through a single confidence threshold, normally estimated on validation data pooled across languages. We ask whether the promise reaches every language, and it does not. On MasakhaNEWS (16 African languages) and AfriSenti (12 languages plus two never seen in training), a pooled threshold meets the 90% target on average but covers Somali at 77.5%, Tigrinya at 83.7%, and the two unseen languages at 77.5% and 81.2%. Estimating one threshold per language brings every language to between 89.1% and 91.0% without retraining, and it shows how unequal the cost of the promise is: keeping it means sending 43% of Somali news and over 80% of Amharic and Xitsonga tweets to a person, against under 8% of Nigerian Pidgin news. One or two hundred labels per language are enough and the models train in minutes on one CPU core, so the fix is affordable: calibrate, report, and budget human review one language at a time.