Search papers, labs, and topics across Lattice.
This paper introduces ReCal3R, a method that calibrates learning rates in streaming 3D reconstruction by assessing the reliability of state tokens. By dynamically adjusting the learning rate based on token alignment, state reconstruction residuals, and update pressure, ReCal3R effectively mitigates the corruption of historical scene information caused by noisy observations. The method demonstrates significant improvements in pose, depth, and reconstruction quality, achieving a 3.7脳 reduction in absolute trajectory error while maintaining efficiency in runtime and memory usage.
Calibrating learning rates based on token reliability can reduce reconstruction errors by over 300% in streaming 3D tasks.
Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7$\times$ reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.
