Search papers, labs, and topics across Lattice.
EdMCGS reconstructs continuous dynamic 3D scenes from extreme-low-frame-rate RGB video paired with asynchronous event streams by modeling Gaussian deformation as an event-driven Markov chain. Unlike prior methods that confine event streams to training-time loss functions, this architecture keeps the state transition active at inference to directly propagate 3D Gaussian motion through time rather than relying on heuristic interpolation. The resulting model outperforms existing RGB- and event-based dynamic novel-view synthesis baselines across synthetic and real-world benchmarks while enabling real-time rendering with significantly fewer Gaussians.
Instead of relegating event data to training-time supervision, dynamic 3D Gaussians can execute active, real-time Markov state transitions directly driven by intervening raw event streams across ultra-sparse RGB keyframes.
We present EdMCGS (Event-driven Markov chain Gaussian Splatting), an end-to-end method for reconstructing dynamic 3D scenes from extreme-low-frame-rate RGB together with an event stream, which can then be rendered at any intermediate timestamp. Methods relying solely on RGB images generate numerous artifacts due to the lack of evidence from between consecutive frames. To supply this missing evidence, we model the scene motion as an event-driven Markov chain, in which the sparse RGB frames anchor the state at their own timestamps while the events recorded within an interval drive the transition across it. Since the transition reads the events of the current interval, it remains active at inference and produces the in-between motion of the 3D Gaussians directly from the events rather than by interpolation, which sets our method apart from prior work that uses events only as training-time supervision. The state is carried by a compact set of control points, each driven by the events sampled in the neighborhood of its own image projection, and a temporal local isometry term keeps the propagated motion locally rigid. Experiments on synthetic and real-world scenes show that EdMCGS outperforms both RGB-based and event-based baselines, while rendering in real time with far fewer Gaussians than the strongest event-based baseline. We release our source code and a new dataset at https://github.com/joseclipse/EdMCGS.
