Search papers, labs, and topics across Lattice.
This paper introduces Motion-Guided Mamba for Video Frame Interpolation (MGMVFI), which enhances State Space Models (SSMs) by employing Motion-Guided Serialization (MGS) to adaptively order inputs based on optical flow. This method allows for improved feature propagation aligned with motion dynamics, addressing the limitations of traditional scanning orders in SSMs. Extensive evaluations reveal that MGMVFI achieves state-of-the-art performance in video frame interpolation, especially for complex motion scenarios, marking a significant advancement in sequential modeling techniques for VFI.
Motion-Guided Mamba redefines video frame interpolation by aligning feature propagation with dynamic motion trajectories, achieving unprecedented accuracy in complex scenarios.
State Space Models (SSMs) have surfaced as a promising architecture in Video Frame Interpolation (VFI), as they can capture long-range dependencies with linear computational complexity. However, their predefined scanning order limits their effectiveness in modeling the dynamic motion trajectories inherent in VFI problems. To tackle this challenge, we propose Motion-Guided Mamba for Video Frame Interpolation (MGMVFI), an adaptation of the selective state space model tailored explicitly for VFI. MGMVFI introduces Motion-Guided Serialization (MGS), which leverages optical flow to define a motion-adaptive 1D input order for the SSM. This aligns the causal state updates with semantically related tokens, enabling motion-consistent feature propagation, particularly for large and dynamic motions. Additionally, to mitigate the unreliable feature representations caused by inaccurate optical flow estimates, we introduce contextual synthesis that utilizes the surrounding spatial context for robust inter-frame feature synthesis. These components are seamlessly integrated within our tailored Mamba architecture, which also employs a lightweight refinement block to enhance local detail reconstruction at a reduced computational cost. Extensive experiments on standard VFI benchmarks demonstrate that MGMVFI achievesstate-of-the-artperformance,particularly on complex and dynamic motions, thereby establishing a new direction for sequence modeling in video interpolation.
