Search papers, labs, and topics across Lattice.
This paper introduces a next-scale autoregressive approach for photorealistic novel view synthesis of human faces, addressing challenges in high spatial resolution and multi-camera outputs. By leveraging lower-resolution pre-training and a smaller, realistic dataset, the model achieves enhanced perceptual fidelity and cross-view coherence without the need for extensive 2D pre-training. The results demonstrate significant improvements in generating sharp, consistent images across multiple viewpoints, showcasing the effectiveness of this architecture for scalable human-centric applications.
Achieving photorealistic human face synthesis with unprecedented cross-view consistency, all while using a smaller training dataset.
Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.
