Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
Gen4P repurposes pretrained video generators into feed-forward 4D perceivers by processing clean video latents in a single encoder pass and predicting 4D attributes via a cross-attention decoder. This on-demand querying framework, enhanced with continuous-time queries and a frozen video-depth LoRA prior, achieves state-of-the-art performance in 3D point tracking, video depth, and camera pose estimation. The approach demonstrates the efficacy of generative pretraining for unified 4D perception from monocular video.
Video generators can be directly transformed into powerful 4D perception systems without iterative refinement, achieving SOTA on tracking and pose estimation.
We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. Together, these results demonstrate the potential of video generative pretraining for unified 4D perception. See https://gen4p.github.io for more visual results.