Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This work tackles the challenge of porting procedural action sequences across visually and structurally heterogeneous environments by conditioning large language models on source and target scene graphs. Cross-environment generalization remains a critical bottleneck for embodied AI, particularly when target spaces have drastically shifted layouts and missing affordances. Evaluated on a new benchmark derived from Ego4D GoalStep, the system successfully synthesizes valid, constraint-satisfying target plans using hierarchical activity abstractions to bridge object-level discrepancies.
Drastic discrepancies in room layouts and object availability no longer derail task execution when LLMs map procedures across hierarchical scene graph abstractions.
We present an action sequence transfer system that adaptively transfers user action sequences across different target spaces. Given an input action sequence from a source space and scene graph representations of both the source and target environments, our system predicts a corresponding action sequence in the target space by adapting to the spatial and object constraints of the new environment. To achieve this, we leverage multi-level representations of user activity to generalize actions at varying levels of abstraction. To demonstrate our system, we collect a new scene graph-based dataset derived from the Ego4D GoalStep dataset for evaluation. Results indicate that our system can generate valid action sequences even between spaces with drastically different object configurations.