Search papers, labs, and topics across Lattice.
This paper explores real-time video game commentary generation using multimodal large language models (MLLMs) via in-context prompting, focusing on the crucial aspect of timing. They introduce two decoding strategies: a fixed-interval approach and a novel dynamic interval-based approach that adjusts prediction timing based on the estimated duration of the previous utterance. Experiments on racing and fighting game datasets demonstrate that the dynamic interval-based decoding generates commentary more aligned with human utterance timing and content, all without fine-tuning.
Forget fine-tuning: Prompting MLLMs with a dynamic interval-based decoding strategy lets them generate surprisingly human-like, pause-aware real-time game commentary.
Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
CMU ML