Search papers, labs, and topics across Lattice.
The paper addresses the challenge of enabling intelligent agents to plan affordance-based interactions with objects in human environments in a manner compatible with human perception and values. They introduce ACKnowledge, a computational framework that integrates a dynamic knowledge graph, a large language model (LLM), and a vision language model (VLM) to reason about physical, intrapersonal, and interpersonal contexts relevant to human-agent interaction. Experiments in real-world simulation tasks demonstrate that ACKnowledge generates acceptable plans with high execution success, significantly improving usage-rights respectfulness and social appropriateness compared to baselines.
Agents can now plan interactions with household objects in a way that's more aligned with human values and social norms, thanks to a new framework that combines knowledge graphs, LLMs, and vision-language models.
Intelligent agents coexisting with humans often need to interact with human-shared objects in environments. Thus, agents should plan their interactions based on objects’ affordances and the current situation to achieve acceptable outcomes. How to support intelligent agents’ planning of affordance-based interactions compatible with human perception and values in real-world contexts remains under-explored. We conducted a formative study identifying the physical, intrapersonal, and interpersonal contexts that count to household human-agent interaction. We then proposed ACKnowledge, a computational framework integrating a dynamic knowledge graph, a large language model, and a vision language model for affordance-based interaction planning in dynamic human environments. In evaluations, ACKnowledge generated acceptable planning results with an understandable process. In real-world simulation tasks, ACKnowledge achieved a high execution success rate and overall acceptability, significantly enhancing usage-rights respectfulness and social appropriateness over baselines. The case study’s feedback demonstrated ACKnowledge’s negotiation and personalization capabilities toward an understandable planning process.
