Search papers, labs, and topics across Lattice.
The paper introduces HiVid, a framework that uses Large Language Models (LLMs) to generate content-aware importance weights for video chunks in both VOD and live streaming scenarios, addressing the limitations of human annotation and vision-saliency models. HiVid incorporates a perception module for local context understanding, a ranking module with LLM-guided merge-sort for VOD, and a prediction module with a multi-modal time series model for low-latency live streaming weight prediction. Experiments demonstrate that HiVid improves weight prediction accuracy by up to 11.5% for VOD and 26% for live streaming, and enhances streaming QoE correlation by 14.7% in user studies.
LLMs can now guide video streaming optimization, outperforming traditional saliency models and human annotation in predicting content importance for both VOD and live streams.
Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.
Tsinghua AI