Search papers, labs, and topics across Lattice.
This paper introduces Agentic Commerce World (ACWorld), a novel environment designed for the evaluation of AI agents in vibe commerce, where users express buying or selling goals in natural language. The Vibe Commerce Protocol (VCP) within ACWorld ensures that agent actions are validated and recorded, enabling auditable and reproducible evaluations of agent interactions in a shared market. Results from the ACWorld Benchmark indicate that agent performance varies significantly across tasks, highlighting the importance of process-level evidence in identifying errors and inefficiencies in agent behavior.
Agentic Commerce World reveals that process-level evidence is crucial for accurately evaluating AI agents in dynamic market environments, challenging traditional reliance on final outcomes alone.
In vibe coding, people describe software in natural language and delegate implementation to AI agents. By analogy, vibe commerce allows people to express buying or selling goals in natural language and delegate the corresponding tasks to agents. Commerce, however, requires independently controlled Buyer and Merchant agents to interact in a shared market while preserving their private objectives and distinct authority. We introduce Agentic Commerce World (ACWorld), an environment for evaluating such agents across ongoing transactions. Through its Vibe Commerce Protocol (VCP), ACWorld validates agent actions before updating shared transaction state and records the resulting interactions, making agent behavior auditable and evaluation reproducible. The ACWorld Benchmark contains a 200-task capability-coverage track and a 60-task large-catalog track that searches 785,022 transactable listings. Across ten models, mean scores range from 65.9% to 85.6% and from 56.1% to 91.4%, respectively. Our analysis shows that process-level evidence is necessary: final state alone can miss evaluated errors, incomplete trajectories still retain useful process signals, and large-catalog tasks expose bottlenecks across stages.
CMU ML