Search papers, labs, and topics across Lattice.
The paper introduces HybridPrompt, a generative video system designed to achieve real-time 1080p decoding on mobile devices by combining generative models for keyframes with traditional codecs for other frames. To address the misalignment between generative and traditional codec paradigms, the authors demonstrate the differentiability of the traditional decoding process, enabling end-to-end optimization. This allows the generative model to synthesize keyframes that are both perceptually high-fidelity and mathematically optimal references for the traditional codec, resulting in a system that outperforms pure neural baselines in speed and achieves improved LPIPS compared to traditional codecs at low bitrates.
Generative video codecs can now run at 150 FPS on smartphones, thanks to a hybrid approach that makes traditional codecs and neural nets work together.
In Video on Demand (VoD) scenarios, traditional codecs are the industry standard due to their high decoding efficiency. However, they suffer from severe quality degradation under low bandwidth conditions. While emerging generative neural codecs offer significantly higher perceptual quality, their reliance on heavy frame-by-frame generation makes real-time playback on mobile devices impractical. We ask: is it possible to combine the blazing-fast speed of traditional standards with the superior visual fidelity of neural approaches? We present HybridPrompt, the first generative-based video system capable of achieving real-time 1080p decoding at over 150 FPS on a commercial smartphone. Specifically, we employ a hybrid architecture that encodes Keyframes using a generative model while relying on traditional codecs for the remaining frames. A major challenge is that the two paradigms have conflicting objectives: the "hallucinated" details from generative models often misalign with the rigid prediction mechanisms of traditional codecs, causing bitrate inefficiency. To address this, we demonstrate that the traditional decoding process is differentiable, enabling an end-to-end optimization loop. This allows us to use subsequent frames as additional supervision, forcing the generative model to synthesize keyframes that are not only perceptually high-fidelity but also mathematically optimal references for the traditional codec. By integrating a two-stage generation strategy, our system outperforms pure neural baselines by orders of magnitude in speed while achieving an average LPIPS gain of 8% over traditional codecs at 200kbps.
