Search papers, labs, and topics across Lattice.
This paper addresses the challenge of functional generalization in robotic tool use, where robots struggle to adapt learned actions to novel tools despite their shared functional intents. The authors introduce FORGE, a two-stage policy that first predicts generalizable keypoint trajectories from action-free data and then grounds these trajectories into robot actions using limited demonstrations. The approach significantly enhances performance, achieving over a 2X improvement in average success rate on a benchmark involving seven different tools, both in simulation and real-world settings.
Robots can now achieve over double the success rate in using novel tools by leveraging keypoint trajectory reasoning for functional generalization.
While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.
