Search papers, labs, and topics across Lattice.
LiveStre4m is introduced as a feed-forward neural network architecture for real-time novel view synthesis from unposed multi-view video, addressing the limitations of optimization-based methods. It uses a multi-view vision transformer for 3D scene reconstruction from keyframes and a diffusion-transformer interpolation module for temporal consistency. By incorporating a camera pose predictor, LiveStre4m achieves an average reconstruction time of 0.07s per frame, enabling real-time NVS streaming with as few as two input streams.
Real-time novel view synthesis from unposed multi-view video is now possible, achieving 38x speedup over previous optimization-based methods.
Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations ($\approx 2.67$s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of $ 0.07$s per-frame at $ 1024 \times 768$ resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m
