Search papers, labs, and topics across Lattice.
This paper addresses the challenge of real-time emotion understanding in streaming video, where predictions must be made from incomplete and evolving evidence. The authors introduce TRACE, a trajectory-aware reliability framework that optimizes low-latency emotion predictions by assessing the reliability of emotion beliefs based on factors such as confidence and class-switching patterns. Experimental results demonstrate that TRACE significantly enhances the accuracy-cost trade-off, achieving high performance while minimizing unnecessary contextual reasoning in dynamic environments.
Real-time emotion predictions can be made more reliable by selectively invoking multimodal reasoning only when necessary, improving accuracy without sacrificing speed.
Video emotion understanding is commonly studied as an offline classification problem, where the complete video segment is available before prediction. Real-time interaction, however, requires emotion decisions from incomplete and evolving evidence. This paper studies streaming video emotion understanding as a reliability-aware decision process over evolving emotion beliefs. In this setting, a single confident prefix prediction can still be unreliable when the underlying belief trajectory is unstable or repeatedly switches across emotion classes. We propose TRACE, a trajectory-aware reliability framework that forms low-latency emotion beliefs from streaming audio prefixes, estimates reliability from confidence, entropy, stability, and class-switching patterns, and selectively invokes contextual belief reinterpretation with visual, textual, and neighboring-utterance evidence. TRACE keeps stable cases in the low-latency online pathway while allocating stronger multimodal reasoning to uncertain cases that remain ambiguous. Experiments on StreamMER, MELD, and MER2024 show that TRACE improves the accuracy-cost trade-off, retaining most full-context gains while reducing unnecessary contextual reasoning.
