Search papers, labs, and topics across Lattice.
VeriTrip, a new benchmark, evaluates travel planning agents' ability to reason over unstructured multimodal web corpora, addressing limitations of existing API-centric benchmarks. It uses a Multimodal Retrieval Base (MRB) and a synchronized Verifiable Knowledge Base (VKB) to assess factual reliability and distinguish reasoning failures from hallucinations. Experiments with leading MLLMs reveal a retrieval-reasoning trade-off, where autonomous retrieval impairs instruction retention.
Autonomous agents struggle to retain instructions when burdened with retrieving information from the open web, exposing a critical retrieval-reasoning trade-off.
Existing benchmarks have laid the foundation for travel planning agents by establishing API-centric paradigms. However, as the capabilities of Autonomous Agents continue to advance, their evaluation must evolve beyond simple tool execution toward handling the inherent complexities of the open web. Current benchmarks bypass core cognitive hurdles: they fail to account for information noise, ignore multi-source factual contradictions, and overlook the necessity of grounding visual perception into logical planning. We introduce VeriTrip, a verifiable benchmark designed to meet the increasing demands for agent robustness and reliability. VeriTrip shifts the evaluation focus to evidence-grounded reasoning over unstructured multimodal web corpora. It establishes a Multimodal Retrieval Base (MRB) derived from real-world sources, forcing agents to autonomously orchestrate queries across heterogeneous data. A synchronized Verifiable Knowledge Base (VKB) enables a cell-wise verification protocol that precisely quantifies factual reliability, distinguishing systematic reasoning failures from parametric hallucinations. Our evaluations across leading MLLMs reveal a critical \textit{retrieval-reasoning trade-off}: the cognitive load of autonomous retrieval significantly erodes instruction retention. VeriTrip provides the rigorous foundation necessary for the next generation of planning agents capable of operating in unconstrained, multimodal environments.
DAMO