Search papers, labs, and topics across Lattice.
This paper introduces IndicContextEval, a comprehensive benchmark designed to assess how effectively Audio Large Language Models (AudioLLMs) utilize contextual information across eight Indic languages. By employing a novel 7-level prompting framework that incorporates various contextual signals, the study evaluates five different models and uncovers significant variances in their ability to leverage context. The findings emphasize the inadequacy of existing benchmarks, revealing that many models do not effectively utilize provided contextual cues, which is crucial for improving their performance in real-world applications.
Evaluating AudioLLMs reveals that many models fail to leverage contextual information effectively, challenging assumptions about their pretraining capabilities.
AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise such context or rely on parametric knowledge learned during pretraining. Existing benchmarks cannot answer this question because they evaluate transcription under fixed prompting conditions and rarely include explicit contextual inputs. We introduce IndicContextEval, a 56-hour multilingual benchmark of natural speech from 555 speakers across 8 Indian languages and 23 professional domains. We design a 7-level prompting framework that progressively introduces contextual signals, including metadata, natural-language descriptions, entity lists in English and native script, and adversarial prompts with incorrect entities. Evaluating five models reveals substantial differences in context utilisation behaviour, highlighting the need for explicit evaluation of contextual grounding in AudioLLMs.
