Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing methods in camera motion understanding by introducing a temporally grounded, compositional recognition framework that accounts for changes in motion within a shot and simultaneous movements. The authors present CamChoreo, a benchmark dataset of 4,229 annotated video clips, which facilitates the training of models to recognize fine-grained camera movements using a compact set of direction-aware labels. They propose CamDistill, a method that distills geometric knowledge into lightweight camera tokens, achieving accuracy comparable to traditional methods without the computational overhead of running a 3D model during inference.
Recognizing complex, simultaneous camera movements in video clips is now feasible without the heavy computational cost of 3D models at inference.
Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.
