Search papers, labs, and topics across Lattice.
To resolve the ambiguity between UAV ego-motion and dynamic scene evolution in aerial video reasoning, the authors develop EgoSIS, a pose-free adapter that factorizes RGB bidirectional optical flow into motion-canonical spatial representations. The architecture tracks ego-transitions via reliability-gated temporal memory, warps visual features to local anchors, and injects four spatial evidence tokens per slice via zero-initialized residuals without inflating base visual token counts. On SIS-Bench, EgoSIS-8B achieves 76.2% overall accuracy and 89.9% perception accuracy, significantly outperforming standard RGB multimodal backbones on self-awareness perception and temporal memory tasks.
Multimodal LLMs can cleanly disentangle camera ego-motion from real-world scene dynamics without any drone telemetry or pose sensors, using just four optical-flow-derived residual tokens per slice to unlock robust aerial video reasoning.
UAV video question answering requires separating camera motion from changes in the scene, but RGB-only multimodal models receive no explicit, stable reference for that separation. We present EgoSIS, a pose-free adapter that converts RGB-derived bidirectional flow into motion-canonical visual evidence in three stages. Factorized Visual Ego-Transitions (FVET) fits a robust image-plane transition and exposes motion, residual-support, and reliability factors. Reliability-Gated Ego-Transition Memory (ReTEM) uses reliability-weighted updates for a bounded history and re-anchors it at cuts or sustained uncertainty. Ego-Aligned Spatial Evidence (EASE) warps supported visual features into each segment's local anchor and injects four spatial evidence tokens per visual slice through zero-initialized residuals, without changing Qwen's visual-token count. On SIS-Bench, EgoSIS-8B obtains 89.9\% perception, 82.5\% perception-plus-memory, and 76.2\% overall accuracy, with the largest gains concentrated in self-awareness perception and memory. The adapter thus provides an interpretable interface between optical flow and spatial reasoning.
