Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces Instance-Anchored Interaction Evidence (IAE) to ground robot plans in human pointing and handling, addressing the challenge of evidence occurring before the final scene and on potentially moved objects. IAE registers scene instances to public identifiers, tracks them via backward mask propagation, and describes frame geometry between human manipulators and these instances. On the WatchAct benchmark, IAE achieves 64.2% plan success on implicit-intent tasks, significantly outperforming a 32B vision-language model (27.5%) and demonstrating robust performance on various manipulation tasks without task-specific training.
Robots can now reliably act on human gestures by anchoring interaction evidence to specific object instances, achieving over 2x success on implicit intent tasks compared to large VLM baselines.
A robot that assists people must often act on what a person has shown rather than said: which of several identical cartons was pointed at, or which box was handled. The plan is executed from the final scene, whereas the evidence occurs earlier, possibly on objects that have since moved. We propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances. An evidence network trained only from task outcomes scores the instances. For pointing tasks, a grammar-constrained dynamic program trained with a structured loss decodes object-destination programs; symbolic programs handle reference disambiguation and, without learning, episodic tasks. On 1,255 WatchAct benchmark requests, scored by symbolic execution, IAE reaches 64.2% plan success on implicit-intent tasks against 27.5% for a 32B vision-language model (strict success 49.7% against 15.4%), and 46.4% against 27.0% on restoration, reversal and imitation without task-specific training. Controls with the same perception overlays, the same 32 frames, forward tracking, or a relation model trained on the same labels do not explain the gain. Given IAE's evidence as text with its meaning explained, the same language model reaches 57.4%: most of the gain comes from the instance-anchored evidence, and the explicit programs add 6.8 points at a fraction of the cost. Pointing remains the hardest case, with 16.9% strict success. The code is available at https://github.com/WeiZhou96/iae-watchact.