Search papers, labs, and topics across Lattice.
EgoPlay is an event-triggered video-to-video editor designed for egocentric streams, fine-tuned on event-conditioned data from Ego4D. By integrating event recognition, temporal restraint, and pixel-level editing into a single end-to-end model, it allows users to specify edits based on events in a monocular video. The system significantly outperforms existing methods, achieving relative gains of up to 17.7% in editing quality and demonstrating superior efficiency with reduced GPU memory usage.
EgoPlay achieves a remarkable 17.7% improvement in editing quality over the previous state-of-the-art, redefining the capabilities of egocentric video editing.
We introduce EgoPlay, an event-triggered video-to-video editor for egocentric streams, obtained by fine-tuning a pretrained V2V diffusion transformer on event-conditioned data built primarily from Ego4D. Given a monocular video and an event-triggered prompt of the form "when X happens, do Y," EgoPlay infers whether and when event X occurs, preserves pre-event frames, and applies edit Y only to the post-event continuation. Rather than cascading a separate event detector with an editor, EgoPlay learns event recognition, temporal restraint, and pixel-level editing jointly in a single end-to-end model, while also handling negative and multi-event prompts. To support this, we construct a large-scale dataset of 106K event-triggered clip-prompt pairs spanning positive triggers, fabricated-trigger negatives, and multi-event prompts. We then train a bidirectional video diffusion editor with event-triggered supervision and derive a causal variant for chunk-by-chunk streamable inference. We further introduce an event-aware evaluation protocol that separately measures post-trigger editing quality, pre-trigger preservation, and false-trigger robustness. On the Ego4D benchmark, EgoPlay substantially outperforms EgoEdit, the state-of-the-art instruction-based egocentric video editing baseline, with relative gains of 17.7%, 16.9%, and 16.4% in editing quality, visual quality, and background consistency. It also surpasses a VLM-guided detector-editor baseline by 15.7%, 14.5%, and 13.5% on the same metrics, while using less than half the GPU memory.
