Search papers, labs, and topics across Lattice.
This paper introduces a Gaussian Process Prior Variational Autoencoder (GPVAE) framework designed for restoring endoscopic videos affected by artifacts such as specular reflections and missing frames. By leveraging a temporal Gaussian process prior, the GPVAE effectively interpolates missing frames while providing uncertainty-aware reconstructions, significantly improving image quality and interpretability for clinicians. Experiments on the C3VDv2 colonoscopy dataset demonstrate that the best GPVAE variants achieve a 21.9% reduction in image reconstruction RMSE and a 12.7% improvement in downstream trajectory accuracy compared to traditional VAE approaches.
Uncertainty-aware reconstruction using a Gaussian process prior leads to a 21.9% improvement in endoscopic video restoration, enhancing clinical interpretability.
Endoscopic video analysis is essential for gastrointestinal diagnosis and computer-assisted interventions, but video sequences are routinely degraded by specular reflections, motion artifacts, and missing frames. These transient corruptions can distract clinicians, reduce image interpretability, and disrupt downstream tasks such as 3D reconstruction and navigation. Effective restoration therefore requires methods that exploit temporal continuity rather than treating frames in isolation. We introduce a Gaussian Process Prior Variational Autoencoder (GPVAE) framework for endoscopic video restoration that replaces the standard factorized latent prior with a temporal Gaussian process prior, enabling interpolation of missing frames with uncertainty-aware reconstruction. The framework combines endoscopy-specific encoders, including a convolutional EndoVAE backbone and pretrained Vision Transformer encoders from GastroNet-5M, with two scalable GP approximations: Hierarchical Prior Approximation (HPA) and Sparse Precision Approximation (SPA). Specular reflections are handled using a DUCKNet-based masking pipeline that excludes corrupted pixels from the reconstruction objective. On the C3VDv2 colonoscopy dataset, the best GPVAE variants reduced image reconstruction RMSE by 21.9\% on average, and by up to 26.1\%, relative to matched VAE baselines. Downstream trajectory RMSE was reduced by 12.7\% on average across classical visual odometry and a pretrained PoseNet, at an average increase of 27.3\% in training time per epoch. Finally, the GP posterior provides per-frame uncertainty estimates that reflect temporal support and offer a confidence signal for restored frames.
