Search papers, labs, and topics across Lattice.
This study revises the RVL-CDIP dataset by identifying and correcting 12% of label errors and addressing approximately 35% test-train overlap, which are critical issues that can skew model performance metrics. The authors create several corrected variations of RVL-CDIP and benchmark document classification performance, revealing that while removing label errors enhances accuracy, eliminating duplicates can negatively impact it. Notably, training on the error-corrected dataset significantly boosts out-of-distribution generalization, with supervised models achieving an average accuracy increase of 8.1 percentage points, and up to 14 points in some cases.
Fixing label errors in RVL-CDIP can enhance model performance by over 8 percentage points, but removing test-train duplicates may paradoxically hurt accuracy.
RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12\% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.
