Search papers, labs, and topics across Lattice.
RenderFormer-V2 casts full global light transport across heterogeneous primitives鈥攊ncluding volumetric media, environment maps, and complex surfaces鈥攁s a generalizable sequence-to-sequence transformer task without requiring per-scene optimization. To scale primitive-to-primitive interactions in the view-independent stage, the architecture pairs windowed attention with domain-specific attention sinks alongside a reflectance-agnostic neural material embedding. The resulting unified feed-forward model synthesizes complex light transport phenomena, such as caustics and volumetric scattering, across diverse and out-of-distribution scenes.
Neural rendering no longer requires per-scene fitting or specialized shaders to capture complex transport effects like caustics and participating media across arbitrary geometries.
We present'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.
