Search papers, labs, and topics across Lattice.
This paper addresses the challenge of evaluating acoustic machine learning systems using partially labeled datasets, which often leads to sampling bias in performance metrics. By employing importance weighting techniques, specifically kernel density estimation, logistic regression, and k-nearest neighbors, the authors aim to correct for this bias under strict annotation-budget constraints. Experiments on an audio scene classification benchmark reveal that these methods significantly improve the accuracy estimates, aligning them more closely with true performance levels.
Importance weighting can dramatically narrow the performance gap in audio classification evaluations, even with limited labeled data.
The performance of acoustic machine learning systems is commonly evaluated using fully annotated test sets. In real-world deployments, however, exhaustively labeling large volumes of continuously collected audio data is often infeasible. Consequently, performance assessment typically relies on a small labeled subset of the available data, introducing a sampling bias that can severely distort evaluation metrics. This paper studies methods for compensating the bias in evaluation-labeled subsets under strict annotation-budget constraints. We study whether importance weighting techniques can mitigate this discrepancy by compensating for the selection bias. Specifically, we implement and compare three density-ratio estimation methods: kernel density estimation (KDE), logistic regression, and k-nearest neighbors (kNN), utilizing feature-space representations of the deployed audio. To emulate realistic deployment scenarios, the labeled subsets are generated using five distinct sampling strategies based on active learning techniques. Experiments conducted on an audio scene classification (ASC) benchmark demonstrate that importance weighting consistently yields more realistic accuracy estimates, significantly reducing the gap between subset-based metrics and the true evaluation performance.
