Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper introduces an Event-guided Neural Video Codec (ENVC) that leverages event streams, which capture brightness changes, to enhance RGB video compression efficiency. ENVC integrates events by generating an event-guided motion prior and an event-conditioned predictor, leading to significant bitrate reductions. Across six benchmarks, ENVC achieved average BD-rate savings of 39.13% (PSNR-RGB) and 67.63% (LPIPS) compared to DCMVC, demonstrating the value of cross-modal guidance.
Event streams unlock nearly 40% BD-rate savings in neural video compression by providing complementary temporal cues for motion and frame prediction.
Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.