Search papers, labs, and topics across Lattice.
This paper introduces RayFormer, a novel transformer-based architecture for NeRF-based video snapshot compressive imaging that leverages patch-level ray sampling to capture content structural similarities. RayFormer models both inter-ray similarities among spatially neighboring points at the same depth and intra-ray correlations between adjacent points along the viewing ray. Experiments demonstrate state-of-the-art reconstruction performance in both simulated and real-world scenes by incorporating a total variation prior to enhance spatial smoothness.
RayFormer achieves state-of-the-art video reconstruction from single snapshots by explicitly modeling inter- and intra-ray similarities, outperforming existing NeRF-based methods.
Video snapshot compressive imaging (SCI) enables the reconstruction of dynamic scenes from a single snapshot measurement. Recently, NeRF-based methods have shown promising reconstruction performance. However, such methods typically adopt random ray sampling strategies and fail to capture content structural similarities, resulting in limited reconstruction quality. To address these issues, we first propose a patch-level ray sampling strategy to enable the modeling of content structure. Then, we propose an Inter- and Intra-Ray Transformer (RayFormer) to capture the structural similarities, modeling both inter-ray similarities among spatially neighboring points at the same depth and intra-ray correlations between adjacent points along the viewing ray. Finally, benefiting from the patch-level sampling strategy, the total variation prior is incorporated into the objective function to enhance spatial smoothness and suppress artifacts. Experiments in both simulated and real-world scenes demonstrate that the proposed method achieves state-of-the-art (SOTA) reconstruction performance.
