Search papers, labs, and topics across Lattice.
Mem3R is introduced as a streaming 3D reconstruction model that uses a hybrid memory design, separating camera tracking (implicit fast-weight memory via a lightweight MLP updated with Test-Time Training) from geometric mapping (explicit token-based fixed-size state). This decoupling addresses drift accumulation and temporal forgetting in long sequences, common in recurrent models with limited latent memory. Experiments show Mem3R improves long-sequence performance and reduces model size compared to CUT3R, achieving up to 39% reduction in Absolute Trajectory Error when combined with TTT3R on sequences of 500-1000 frames.
By decoupling camera tracking and geometric mapping with a hybrid memory design, Mem3R achieves state-of-the-art streaming 3D reconstruction while shrinking model size by 19%.
Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from drift accumulation and temporal forgetting over long sequences due to the limited capacity of compressed latent memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, integrating it with TTT3R decreases Absolute Trajectory Error by up to 39% over the base implementation on 500 to 1000 frame sequences. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction, while preserving constant GPU memory usage and comparable inference throughput. Project page: https://lck666666.github.io/Mem3R/
