Search papers, labs, and topics across Lattice.
This paper introduces UniD, a unified video model that predicts eight dense scene properties using disjoint, domain-specific datasets without requiring annotation overlap or pseudo-labeling. By leveraging a distillation step where per-task experts guide a unified backbone through lightweight task projectors, the model effectively utilizes strong visual priors from a pretrained diffusion model to enhance generalization across various scene-task combinations. The results show that UniD achieves competitive performance compared to specialized models and multi-task approaches, demonstrating improved robustness in out-of-distribution scenarios and better temporal and cross-task consistency.
UniD achieves competitive performance in dense scene prediction by effectively bridging domain gaps without the need for overlapping annotations or costly pseudo-labeling.
Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.
