Search papers, labs, and topics across Lattice.
OmniStream is introduced as a unified streaming visual backbone designed for real-time environments, integrating perception, reconstruction, and action capabilities. The model uses causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE) to enable efficient, frame-by-frame online processing of video streams with a persistent KV-cache. Pre-training on 29 datasets using a multi-task framework demonstrates competitive performance across various tasks, including image/video probing, geometric reconstruction, video/spatial reasoning, and robotic manipulation, even with a frozen backbone.
A single visual backbone can now handle perception, reconstruction, and action in streaming video, rivaling specialized models without task-specific fine-tuning.
Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.
