Search papers, labs, and topics across Lattice.
This paper re-evaluates DistilHuBERT's accent classification robustness by employing speaker-disjoint splits and cross-corpus generalization on L2-ARCTIC and external English accent datasets, revealing that standard in-corpus evaluations significantly overestimate performance. While balanced external-data training shows promise, models trained solely on external data fail to generalize reliably to L2-ARCTIC, and confidence-based pseudo-labeling proves insufficient for domain mismatch. The findings underscore the necessity of concurrent speaker-disjoint and cross-corpus evaluations for accurate robustness assessment.
Standard accent classification evaluations overstate model robustness by up to 40% due to speaker overlap; speaker-disjoint, cross-corpus testing reveals a stark performance drop.
Recent accent classification systems can achieve high accuracy in standard in-corpus evaluations. However, when speakers overlap between the training and test sets, such results may overestimate model robustness compared with real-world deployment scenarios. This study revisits the robustness of DistilHuBERT-based accent classification by using L2-ARCTIC and multiple external English accent datasets. The comparison covers file-level and speaker-disjoint splits, along with balanced external-data training and teacher-student pseudo-labeling. The large performance gap between in-corpus and external evaluations suggests that file-level evaluation substantially overestimates performance, while speaker-disjoint evaluation can provide a more realistic estimate of the generalization to unseen speakers. In this study, models trained on external data alone do not reliably generalize to L2-ARCTIC. Among the tested configurations, balanced external-data training offers the most promising supervised setting. The pseudo-labeling experiments further indicate that confidence-based filtering alone is insufficient under domain mismatch. Overall, this study suggests that robust accent classification evaluation should simultaneously include speaker-disjoint evaluation and cross-corpus testing.
