Search papers, labs, and topics across Lattice.
This paper investigates the impact of training data quality on classifier performance in the context of metagenomic assembly, focusing on the degradation of training data through multiple mechanisms. The study employs four classifiers—Bayes classifiers, neural networks, partition models, and random forests—to assess both individual and congruent behavior under varying levels of data degradation. The key finding reveals a breakdown-like behavior across all classifiers, transitioning from accurate classification to coincidental correctness as training data quality diminishes, highlighting the critical role of training data quality in classifier efficacy.
Classifiers trained on degraded data don't just become inaccurate, they become *coincidentally* accurate, failing in similar ways and masking the underlying problem.
We describe extensive numerical experiments assessing and quantifying how classifier performance depends on the quality of the training data, a frequently neglected component of the analysis of classifiers. More specifically, in the scientific context of metagenomic assembly of short DNA reads into "contigs," we examine the effects of degrading the quality of the training data by multiple mechanisms, and for four classifiers -- Bayes classifiers, neural nets, partition models and random forests. We investigate both individual behavior and congruence among the classifiers. We find breakdown-like behavior that holds for all four classifiers, as degradation increases and they move from being mostly correct to only coincidentally correct, because they are wrong in the same way. In the process, a picture of spatial heterogeneity emerges: as the training data move farther from analysis data, classifier decisions degenerate, the boundary becomes less dense, and congruence increases.
