Search papers, labs, and topics across Lattice.
This paper introduces the Analogical Deep Research (ADR) task for Large Language Models (LLMs) and presents the ADR-bench benchmark to evaluate LLMs' ability to retrieve and integrate historical analogies for foresight analysis. The authors identify a critical limitation in existing LLMs: their tendency to focus on surface features rather than the underlying causal mechanisms of events, which hinders effective analogy generation. To address this, they propose the Causal Analogical Researcher (CANA) framework, which enhances analogy identification and integration through structural decomposition and feedback, resulting in up to 10% improvement over state-of-the-art methods on the ADR-bench.
LLMs struggle with historical analogy retrieval due to a focus on superficial similarities, but the CANA framework significantly enhances their performance by emphasizing causal understanding.
Systematic comparisons between current situations and structurally similar past events in the historical, i.e., historical analogies, is among the most powerful tools for foresight analysis. In this work, we present a new task called Analogical Deep Research (ADR) to Large Language Model (LLM) agents and construct the first ADR benchmark ADR-bench to study whether LLM agents are able to find and leverage historical analogies when doing foresight analysis. Our investigation reveals a key obstacle: LLM agents are poor at finding analogies because they match on surface features rather than underlying mechanisms. We argue that ADR is inherently a causal question as it requires understanding why the event occurred. Based on our theoretical analysis, we propose two principles required for ADR, including the mechanism alignment and cross-analogy confirmation. Built upon our theoretical results, we propose a new agentic framework called Causal Analogical Researcher (CANA) that guides LLMs to find and integrate historical analogies. CANA incorporates a simple yet effective structural decomposition representation, and integrates structural feedback for reflective improvements of historical analogy identification and integration. We show that CANA brings up to 10% improvements in historical analogy generation, and surpasses the state-of-the-art deep research agents in the ADR-bench. Case studies with the ongoing events confirm the effectiveness of CANA in leveraging historical analogies.
CMU ML