Search papers, labs, and topics across Lattice.
This paper introduces a dual-disentangling feature extraction framework that effectively separates appearance and motion information from both image and event modalities, addressing the limitations of current multimodal moving instance segmentation (MIS) methods. By implementing a multi-granularity cross-modal alignment, the approach enhances the fusion of spatial and temporal details, particularly benefiting the segmentation of small moving instances in challenging environments. Experimental results show that this method achieves state-of-the-art performance in MIS, outperforming existing techniques, especially in scenarios involving fast motion and low-light conditions.
Small moving instances can now be segmented more effectively than ever, even in low-light and fast-motion scenarios.
Moving instance segmentation (MIS) attracts increasing attention due to its broad applications in traffic surveillance, autonomous driving, and animal tracking. Event cameras record asynchronous brightness changes, providing high temporal resolution and dynamic range, which makes them highly sensitive to motion information. By fusing event and image features, motion cues from events can complement spatial details from images, enhancing the performance of MIS. However, current multimodal MIS methods still struggle to segment small moving instances, as event cameras often yield sparse features under limited resolution. Moreover, event features entangle appearance attributes with motion cues, which further restricts effective cross-modal fusion. To address these challenges, we first propose a dual-disentangling feature extraction framework that separates and extracts appearance and motion information within both image and event modalities, thereby improving feature density. Subsequently, a multi-granularity cross-modal alignment is introduced to align distributionally and semantically consistent features across modalities, enabling more effective fusion with rich spatial and temporal details. The experiment results demonstrate that our method achieves state-of-the-art performance in multimodal MIS, especially for small instances under challenging conditions such as fast motion and low-light settings.
