Haolin He

Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction, which leads the compared models on MMAU, Llama Questions, and Speech CMMLU.

LALMs face significant challenges in audio comprehension, especially in extracting relevant facts from lengthy signals, with performance deteriorating as audio duration increases.

Audio-language models struggle significantly with hidden evaluation tasks, with accuracy plummeting by nearly 12 percentage points on average, highlighting the challenge of true audio comprehension.

Achieving one-step audio waveform generation with a 17× speedup while preserving quality could revolutionize TTS systems.

Unlock SOTA audio understanding by jointly training on readily available clip-level descriptions and scarce frame-level annotations, bridging the gap between global semantics and local details.