Search papers, labs, and topics across Lattice.
The paper introduces PhysMLLMs, a novel architecture that integrates physics-inspired spatial continuity priors into video multimodal large language models (MLLMs) to address spatio-temporal inconsistencies such as jitter and identity switches during video segmentation. By employing a Global Representation Prior Alignment (REPA-Global) mechanism, the model aligns global visual representations from a frozen teacher model, enhancing object-centered stability without increasing inference time. Experimental results demonstrate significant improvements in segmentation quality and cross-frame consistency, particularly in challenging scenarios, while maintaining performance in single-frame image segmentation tasks.
Injecting physics-inspired spatial priors into video MLLMs drastically enhances temporal stability and segmentation accuracy without sacrificing image-level performance.
Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git.
