Search papers, labs, and topics across Lattice.
This paper introduces LIME, a novel framework for generating language-conditioned camera motion in autonomous robots by predicting relative target camera poses based on user intent expressed in natural language. By leveraging egocentric video to mine multi-intention camera-motion supervision, LIME effectively captures the complex relationship between perceptual intentions and camera movements across various semantic levels. The results demonstrate that LIME can successfully translate passive human video into actionable insights for intent-aware active perception, enhancing the robot's ability to adapt its viewpoint in response to user instructions.
LIME turns ordinary egocentric video into a powerful tool for robots to dynamically adjust their camera poses based on user intent, revolutionizing how we think about robotic perception.
Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, language-conditioned camera motion remains comparatively underexplored as a first-class action. We formulate language-conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. This task is inherently non-trivial: viewpoint changes are driven by latent perceptual intentions, and a valid motion may operate at different semantic granularity, from entering a room to looking around a corner, inspecting a visible object, or revealing an occluded detail. To model this structure, we mine multi-intention camera-motion supervision from egocentric video, pairing plausible intents and observation-gain descriptions with relative SE(3) target poses. We propose LIME, a vision-language camera-motion generator that combines an auto-regressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across experiments and downstream robotic tasks, we show that LIME can learn to actively choose camera poses from passive human video, turning ordinary egocentric recordings into supervision for intent-aware active perception.
ETH