Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
RealDenseFace introduces a real-time optimization-based method for monocular 3D face reconstruction that leverages dense UV-space predictions from a neural network. By reformulating the fitting of a 3D Morphable Model (3DMM) as a nonlinear least-squares problem and employing a tailored Gauss-Newton solver, the method achieves rapid convergence in just a few iterations. This approach not only delivers state-of-the-art accuracy on the NeRSemble SVFR benchmark but also enables online tracking at over 80 FPS and offline sequence reconstruction that is more than 20 times faster than existing methods.
Achieving real-time 3D face reconstruction at over 80 FPS while maintaining state-of-the-art accuracy could revolutionize applications in virtual reality and telepresence.
Recent monocular 3D face reconstruction methods achieve high fidelity by fitting a 3D Morphable Model (3DMM) to dense priors predicted by networks, but the optimization stage is computationally expensive, often taking tens of seconds per image. We present RealDenseFace, a real-time optimization-based 3D face reconstruction method with dense UV-space network predictions. Our key idea is to formulate 3DMM fitting as a nonlinear least-squares problem and solve it with a tailored Gauss-Newton solver that converges in only a few iterations. The reconstruction is conducted in two stages. In the first stage, the network predicts two dense UV-space maps from a single RGB image: a correspondence map for UV-to-image alignment, and a relative-depth map for geometric constraints along the viewing direction. In the second stage, the solver fits per-vertex targets sampled from these maps at the vertex UV coordinates. The solver supports all three reconstruction settings: single-image fitting, offline sequence reconstruction, and online tracking. Our method achieves state-of-the-art accuracy on the NeRSemble SVFR benchmark. The online tracker runs at 80+ FPS, and the offline sequence reconstruction is over 20 times faster than previous optimization-based baselines.