Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a deep neural timbre trait predictor that combines a pretrained audio neural embedding (CLAP) with a shallow learnable component, trained on the RWC musical instrument database and human assessments of 20 timbre descriptions. The model demonstrates a strong correlation with human ratings (r = 0.66, p<0.001), providing a more interpretable assessment of musical sound synthesizers compared to traditional metrics like the Fréchet Audio Distance (FAD). By applying this predictor to evaluate TokenSynth, the authors show it can rank synthesizer performance equivalently to FAD while also identifying specific timbral dimensions for improvement in generated sounds.
A deep learning model that not only ranks synthesizer performance but also reveals actionable insights into timbral quality, bridging the gap between numerical metrics and human perception.
Measuring neural audio synthesizers'performance is now routinely conducted using distribution based metrics such as the Fr\'echet Audio Distance (FAD). Although this metric can be correlated with human perception, it offers limited interpretability beyond ranking different approaches. In this paper, we introduce a deep neural timbre trait predictor composed of a pretrained audio neural embedding (CLAP), and a shallow learnable component. The latter is trained using the RWC musical instrument database and human judgments of 20 timbre descriptions (e.g., woody, percussive, rumbling, etc.) for 31 instruments. The resulting model shows strong correlation with average human ratings (r = 0.66, p<0.001). We then demonstrate the benefit of this predictor for evaluating the performance of TokenSynth, a neural sound synthesizer. First, the Mean Absolute Error (MAE) computed over the set of generated sounds under different conditioning modalities of the model provides the same ranking as a FAD computed with the RWC database as a reference, suggesting that the proposed predictors are able to provide equivalent information on a distributional basis. Second, because the model is able to qualitatively analyze isolated sounds, we can determine which generated sounds could be improved and identify specific timbral dimensions that need adjustment.