Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces LEGO, a lifting-free approach for exocentric-to-egocentric video generation that bypasses explicit scene reconstruction. Instead, it employs a learned view synthesizer (LVSM-style transformer) to directly render the egocentric view by learning cross-view correspondences probabilistically. This method outperforms state-of-the-art explicit pipelines by prioritizing structural alignment over sharp texture, with the diffusion model then restoring detail.
Egocentric video generation can be achieved by a learned transformer that probabilistically synthesizes novel views, outperforming explicit depth-based reconstruction pipelines.
Generating an egocentric video from a single exocentric recording is a challenging case of novel view synthesis, as the two cameras share little overlap and much of the target view is unobserved. Current state-of-the-art methods reconstruct the scene explicitly by estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. This deterministic mapping assigns each pixel to a single reprojected location, which preserves texture but translates depth errors into misplaced content. We ask what a video diffusion model should receive as its condition and propose a lifting-free answer: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly without depth, point clouds, or reprojection, resolving cross-view correspondence internally. In contrast, its probabilistic mapping averages each region over candidate source locations according to a learned correspondence distribution, preserving structure while fine texture is averaged away. We argue that this trade-off suits a diffusion generator, whose denoising training excels at restoring detail, so an effective condition should prioritize structural alignment over sharpness. This distribution's concentration also yields a per-region confidence, used both to mask low-confidence regions and to guide the generator toward high-confidence areas during early layout-forming denoising steps. Our approach consistently outperforms the state-of-the-art explicit pipeline and generalizes to other datasets without retraining. The synthesizer thus supplies view structure, and the diffusion model its detail.