Search papers, labs, and topics across Lattice.
This paper investigates the generalization capabilities of multimodal foundation models fine-tuned on diverse interactive assistance data for open-set corrective assistance in a synthetic Overcooked environment. They evaluate generalization along two axes: unseen categories of user behavior and new configurations not encountered during training. The results show that performant models benefit from datasets covering multimodal grounding, defect inference, and exposure to diverse scenarios, providing key insights into the nature of assistive datasets required for open-set assistive intelligence.
Open-set corrective assistance, requiring models to inspect lengthy user behavior and provide corrective actions or language-based feedback, remains a significant challenge even with fine-tuning on diverse interactive data.
Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called \textbf{Open-Set Corrective Assistance}, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach provides key insights into the nature of assistive datasets required to enable open-set assistive intelligence. In particular, we show that performant models benefit from datasets that cover different aspects of assistance, including multimodal grounding, defect inference, and exposure to diverse scenarios.
