Search papers, labs, and topics across Lattice.
This paper introduces SENTRY, a novel module that enhances memory update mechanisms in SAM2-based visual object tracking by validating memory updates for temporal consistency before they are committed. By employing neighbor-aware cycle-consistent matching and aggregating diverse segmentation hypotheses, SENTRY effectively mitigates drift caused by occlusion and rapid motion. The integration of SENTRY into existing tracking architectures yields significant performance improvements, achieving state-of-the-art results across multiple benchmarks without requiring retraining.
Enforcing temporal validity during memory updates can stabilize visual tracking systems and lead to unprecedented performance gains across nine benchmarks.
We revisit the memory update mechanism in SAM2-based visual object tracking and identify confidence-only mask selection as the dominant cause of drift under occlusion, rapid motion, and distractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizon temporal consistency before committing it. SENTRY aggregates diverse segmentation hypotheses per frame, backtracks them into short tracklets, and uses neighbor-aware cycle-consistent matching against recent trajectories to favor temporally and geometrically consistent masks. It leaves the base architecture untouched, replacing confidence-driven writes with consistency-validated ones. For fair evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, filling gaps in prior reports. Integrated into five strong baselines, SENTRY delivers consistent gains across nine benchmarks, achieving new zero-shot SOTA on LaSOT, LaSOT_ext, GOT-10k, VOT20, VOT22, and DiDi. Despite these checks, the SAM2-L version runs at 32.8 FPS on an A100, and across compatible hosts adds only about 0.4--0.6 GB VRAM. Our results provide the first unified all-scale evaluation of SAM2-based trackers and show that enforcing temporal validity at write time stabilizes memory-augmented tracking without retraining.
