Search papers, labs, and topics across Lattice.
This study benchmarks eleven audio classification methods, including five task-aware closed-set LLMs and various traditional classifiers, on a closed-set sound-source identification task using 2,242 audio clips across 23 classes. The evaluation employs a tiered approach to account for the differing methodologies and scoring systems, revealing that the Gemini-3.1-Pro-Preview model achieves the highest category-level F1 score of 85.6%. Notably, while some models excel in category-level identification, they struggle with fine-grained classification, highlighting significant structural error modes and offering practical guidance for method selection.
The Gemini-3.1-Pro-Preview model outperforms traditional classifiers in sound-source identification, but even top models confidently produce incorrect answers 92-100% of the time.
We benchmark eleven audio classification methods: five task-aware closed-set LLMs (four Gemini models plus open-weight Kimi-Audio-7B-Instruct), four fixed-vocabulary taggers (YAMNet, PANNs, Whisper-AT, and SSLAM), a zero-shot audio-text model (CLAP), and an audio-grounded LLM (BAT). We evaluate them on a closed-set sound-source identification task over 2,242 clips spanning 23 fine-grained classes and 11 categories. Since these methods differ fundamentally in how they receive the task and how outputs are scored, we group them into four evaluation tiers rather than one leaderboard, reporting macro Precision, Recall, F1, and false-negative rate per tier. The best model, Gemini-3.1-Pro-Preview, reaches 85.6 percent category-level F1 and 56.7 percent fine-grained F1. Kimi-Audio is competitive for its size, reaching 67.5 percent category-level F1 and 32.9 percent fine-grained F1, but fails to answer 1.6 percent of samples. SSLAM and CLAP match or exceed the best closed-set model at the category level without seeing the candidate list, but fall behind at the fine-grained level. Analyzing the Gemini models'chain-of-thought across 8,968 responses, we find that response length does not predict accuracy, an apparent"holistic judgment beats detailed analysis"effect is better explained as a difficulty confound, and wrong answers are stated confidently 92 to 100 percent of the time. We report full per-class confusion matrices and metrics for all eleven methods, identify the structural error modes behind most of the accuracy loss between granularities, and give practical guidance for choosing among these method families.
