Search papers, labs, and topics across Lattice.
ProxyPose addresses the challenge of 6-DoF pose tracking from monocular video by reformulating the problem as video-to-video translation, requiring only a single marked pixel in the first frame. By employing a fine-tuned video diffusion model, it generates a synthetic proxy video that accurately represents the motion of the object, enabling classical pose estimation techniques to recover the full trajectory. This approach not only achieves state-of-the-art accuracy without relying on additional inputs but also extends its applicability to face tracking and camera pose estimation in complex real-world scenarios.
Achieving state-of-the-art 6-DoF pose tracking accuracy without the need for 3D models or depth maps, ProxyPose redefines the landscape of monocular video tracking.
Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. To tackle this problem, existing methods require inputs beyond the video itself-such as 3D models, depth maps, object masks, or task-specific learned features-and they struggle with textureless, transparent, reflective, or deformable surfaces. Here, we introduce ProxyPose, which recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video-a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classical pose estimation with off-the-shelf solvers. This formulation leverages large-scale video pre-training to absorb the hardest aspects of pose tracking-handling challenging materials, occlusions, and deformations-into the translation step, while operating at the pixel level with no assumptions about object identity, boundaries, or global rigidity. ProxyPose achieves state-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods and after fine-tuning the video model only on synthetic data. We further demonstrate that ProxyPose extends to face tracking, camera pose estimation, and challenging in-the-wild scenes that are beyond the reach of existing approaches. Project page: https://ruihangzhang97.github.io/proxypose/.
