Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper compares Confident Learning and Dataset Cartography for automated label error detection in Russian text classification datasets. They fine-tuned RuBERT on three datasets (RuCoLA, TERRa, ru_emotion_e-culture) and evaluated the impact of filtering noisy data points identified by each method. Confident Learning improved performance on small, noisy datasets, while both methods outperformed random data removal, suggesting they effectively identify problematic examples.
Turns out, how well your data filtering works depends a LOT on the size and noise level of your dataset.
Data quality is a critical factor in the effectiveness of machine learning models. Label errors, present even in widely used benchmarks, introduce noise into training data and reduce model generalization. In this work, we conduct a comparative analysis of two automatic label error detection methods - Confident Learning and Dataset Cartography - on three Russian text classification corpora of varying size, number of classes, and domain: ru_emotion_e-culture (49,123 examples, emotion classification), RuCoLA (8,524 examples, linguistic acceptability), and TERRa (2,337 examples, textual entailment recognition). We use the pre-trained rubert-base-cased model fine-tuned on each corpus. To verify the meaningfulness of filtering, we conduct control experiments with random removal of an equivalent number of examples. Results show that the effectiveness of both methods depends strongly on dataset characteristics: on large corpora with low noise levels, filtering does not improve performance, while on small datasets with high noise, Confident Learning achieves a significant F1-macro improvement. Dataset Cartography demonstrates more conservative behavior, removing fewer examples. Across all corpora, targeted removal by both methods outperforms random removal, confirming the meaningfulness of the approaches.