Search papers, labs, and topics across Lattice.
This paper introduces ENCORE, a novel framework for learned video compression that leverages event camera data to enhance motion estimation in traditional RGB-based codecs. By employing Complementary Motion Representation (CMR) to separate motion features and utilizing Spatial Energy and Redundancy-Informed Calibration (SERIC) for flow correction, ENCORE effectively mitigates the limitations of existing methods that struggle with rapid motion and other challenging conditions. The results show significant improvements in compression efficiency, achieving up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings on benchmark datasets, demonstrating the effectiveness of integrating event data into video compression workflows.
Event cameras can boost video compression performance, yielding up to 22% better efficiency by refining motion estimates in challenging conditions.
Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.
