Search papers, labs, and topics across Lattice.
PARSA-Bench, a new benchmark, is introduced to evaluate audio-language models on Persian language and culture, addressing gaps in existing benchmarks. The benchmark comprises 16 tasks, including 10 newly introduced tasks focused on poetry, music, and code-switching, with over 8,000 samples. Results show that text-only baselines outperform audio counterparts and models struggle with culturally-grounded tasks like vazn detection, indicating limitations in leveraging audio-specific information and prosodic perception.
Current audio-language models are culturally tone-deaf, failing to grasp nuances of Persian poetry and music despite strong performance on transcribed text.
Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching - none captured by existing benchmarks. We introduce PARSA-Bench (Persian Audio Reasoning and Speech Assessment Benchmark), the first benchmark for evaluating large audio-language models on Persian language and culture, comprising 16 tasks and over 8,000 samples across speech understanding, paralinguistic analysis, and cultural audio understanding. Ten tasks are newly introduced, including poetry meter and style detection, traditional Persian music understanding, and code-switching detection. Text-only baselines consistently outperform audio counterparts, suggesting models may not leverage audio-specific information beyond what transcription alone provides. Culturally-grounded tasks expose a qualitatively distinct failure mode: all models perform near random chance on vazn detection regardless of scale, suggesting prosodic perception remains beyond the reach of current models. The dataset is publicly available at https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench
