Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HICO-Bench, a novel benchmark for evaluating social agents that assesses both conversational integrity and plan feasibility via verifiable outcomes in paired coordination scenarios. The benchmark reveals that current LLMs often fail to achieve Verified Success (passing both integrity and feasibility checks), with 36% of conversations rated highly for goal completion failing this stricter criterion. To address this, the authors propose Paired-Credit Policy Optimization (PaCPO), a method that leverages outcome feedback from paired scenarios to improve agent performance, increasing Verified Success for Qwen3-8B from 20.4% to 28.3%.
Most social LLMs fail to achieve verifiable success in coordination tasks, even when rated highly by conversational judges.
Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.
DAMO