Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing audio-language models, specifically the Querying Transformer (Q-Former) connector, which suffers from output vector collapse and loss of paralinguistic cues. The authors introduce ORCA, a method that groups queries to ensure diverse directional outputs, significantly enhancing model performance. ORCA achieves a remarkable 26.4-point improvement on the SAKURA multi-hop reasoning task, outperforming larger models while also reducing query redundancy and increasing cross-speaker variance.
ORCA not only boosts performance by 26.4 points but also restores critical speaker identity cues that traditional models overlook.
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
