Search papers, labs, and topics across Lattice.
This paper addresses the challenge of online novel view synthesis from multi-view streaming videos by proposing a novel approach that decouples memory update frequencies from memory application. By utilizing periodic memory updates and applying memory on a per-frame basis with cross-view attention, the authors effectively manage dynamic scene deformations while maintaining real-time performance. Key results show that their method achieves state-of-the-art performance in synthesizing views of dynamic human motion and enables minute-scale online memorization without sacrificing stability or efficiency.
Real-time novel view synthesis is now achievable without the computational burden of constant memory updates, thanks to a clever decoupling strategy.
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts. Given that memory updates are more demanding than memory application and video content is largely redundant, we propose to decouple the frequencies of these two processes. Our approach performs periodic memory updates while applying the memory on a per-frame basis, using cross-view attention to manage deformations between the prior memory state and the current frame. To lock in the historical context, we introduce two critical mechanisms: an auxiliary Memory Loss that forces persistent internalization of the scene, and a Memory Caching strategy that regularizes active weights against catastrophic drift. Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.
