Search papers, labs, and topics across Lattice.
EgoRelight is presented as a novel framework for egocentric telepresence that captures full-body human performance, synthesizes photorealistic and relightable appearance, and estimates HDR environment maps from a single HMD. It uses stereo down-facing cameras for dense depth map extraction to drive a mesh-based avatar and introduces a neural appearance model that learns view-dependent specular and view-independent diffuse shading separately using a specialized ray-sampling strategy. The system integrates avatars into physical environments via test-time inverse rendering to recover HDR environment maps, demonstrating superior performance in geometric accuracy, rendering, and relighting fidelity compared to state-of-the-art baselines.
Imagine telepresence where your avatar convincingly blends into any environment, relit in real-time based on the scene's actual lighting, all from a single headset.
Mixed Reality (MR) headsets promise a future of immersive telepresence where virtual humans blend indistinguishably into real or virtual surroundings. Achieving this vision requires a method for capturing a user's motion, estimating appearance under novel lighting, and understanding the environment - all from the constrained viewpoint of a head-mounted display (HMD). Existing approaches treat these as isolated problems: they either focus on driving avatars with baked-in lighting or rely on studio setups for relighting. In this paper, we present EgoRelight, a holistic framework for egocentric telepresence that simultaneously captures full-body human performance, synthesizes photorealistic and relightable appearance, and estimates high dynamic range (HDR) environment maps from a single HMD. First, to ensure motion and surface reconstruction, we propose an egocentric perception module that leverages stereo down-facing cameras to extract dense depth maps, which serve as geometric control signals to drive a mesh-based avatar. Second, we introduce a novel neural appearance model that learns to synthesize view-dependent specular and view-independent diffuse shading separately. By employing a specialized ray-sampling strategy, our model generalizes to unseen illumination without relying on restrictive analytical BRDF priors. Third, we enable seamless avatar integration into the physical world via a test-time inverse rendering process, which recovers an HDR environment map by matching the pre-trained avatar's appearance to live egocentric camera observations. We demonstrate our system through a social telepresence application, where remote users are coherently relit according to their physical environment. Extensive experiments show that our components and the integrated system significantly outperform state-of-the-art baselines in geometric accuracy and rendering as well as relighting fidelity.
Google Research