Search papers, labs, and topics across Lattice.
This paper introduces Instance-Oriented Memory (IOM), an innovative framework that enables robots to efficiently manipulate objects with hidden internal states by recording and reusing manipulation procedures based on identifiable object features. Unlike traditional memory systems that focus on task success, IOM allows for amortization of exploration costs by recalling learned procedures from a single encounter, significantly reducing the number of manipulation operations required. Experimental results demonstrate that IOM can cut manipulation operations by 16-30% while maintaining high success rates, even when utilizing an off-the-shelf vision-language model without task-specific training.
IOM enables robots to recall manipulation procedures from a single encounter, slashing operation counts by up to 30% while ensuring task success remains intact.
Manipulating objects with hidden internal state, such as a latched microwave, forces a robot to probe before it can act. Yet a robot that has solved an instance once re-runs the same probes whenever it encounters that instance again, because existing cross-episode memories target task success and organize reuse around states, not the object or the cost of re-exploring it. We present Instance-Oriented Memory (IOM), an object-centric framework that amortizes this exploration: from a single encounter that uncovers the hidden state, whether or not it succeeds, IOM records a short procedure for manipulating that instance, keys it on the object's identifiable features, and injects it as a soft bias on a procedure-conditioned policy. A later encounter recognizes the object and recalls its procedure instead of re-exploring. We instantiate this distillation with an off-the-shelf vision-language model (VLM) that parses each encounter into the procedure without task-specific training. Across four articulated-object tasks, two in simulation (microwave, door) and two on a real robot (bottle, cabinet), an oracle procedure memory cuts manipulation operations by 16-30% over re-exploration at non-regressing success, and the VLM instantiation recovers 69-88% of that saving out of the box. Because the procedure is a soft bias on a feedback-driven policy, an incorrect memory is recovered from rather than obeyed: success holds even when a retrieved procedure is wrong, as for $\approx$12% of door instances. Across all tasks the benefit is purely one of efficiency: success never regresses, and on the real robot even improves. Code will be released upon acceptance.
