Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Gekko, a self-supervised pre-training network that enhances 3D vision feature learning by leveraging reconstruction error comparisons between co-visible and non-co-visible regions in image pairs. By transforming the limitations of monocular training signals into a useful co-visibility proxy, Gekko achieves significant performance improvements over existing methods like CroCo, demonstrating up to 6 times higher accuracy in zero-shot correspondence estimation and a 22% reduction in endpoint error on the ETH3D dataset. The model's ability to detect co-visibility in unseen scenes and its compatibility with raw video training streamline the process, eliminating the need for extensive 3D preprocessing.
Gekko turns the challenge of monocular signals into a powerful co-visibility detector, achieving unprecedented accuracy in 3D vision tasks without the need for ground-truth annotations.
Self-supervised pre-training via cross-view completion learns strong features for 3D vision from co-visible regions of image pairs. However, the reference view provides little information for reconstructing non-co-visible patches, implicitly yielding a monocular training signal in these regions. We introduce Gekko, which turns this limitation into a useful signal. The relative improvement of the cross-view reconstruction error over a masked-autoencoder error is a self-supervised proxy for co-visibility: large improvements indicate co-visible regions, negligible ones non-co-visible areas. Gekko is a network, trained from scratch, that jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of this relative improvement, providing an additional binocular signal for all masked regions without any ground-truth 3D annotation. Under identical architectures and training data, Gekko consistently outperforms CroCo on zero-shot correspondence estimation, relative pose estimation, and pointmap regression, with up to 6 times higher accuracy at the strictest relative-pose threshold and a 22% drop in end-point error on ETH3D. The extra channel it learns is itself a strong co-visibility detector on unseen scenes, and Gekko's frozen features outperform released cross-view backbones of comparable or larger size. It can also be trained directly from raw videos with a simple stride-based curriculum, removing the cumbersome 3D preprocessing prior methods require while matching models trained on curated data. Code and pre-trained models are publicly available.