Search papers, labs, and topics across Lattice.
This paper introduces the Analogical Deep Research (ADR) task for Large Language Models (LLMs) and establishes the ADR-bench benchmark to evaluate LLMs' ability to identify and utilize historical analogies for foresight analysis. The authors identify a significant limitation in existing LLMs, which tend to focus on superficial features rather than the underlying causal mechanisms of historical events. By proposing the Causal Analogical Researcher (CANA) framework, which emphasizes mechanism alignment and cross-analogy confirmation, the authors demonstrate a 10% improvement in analogy generation compared to state-of-the-art models, showcasing CANA's effectiveness in real-world case studies.
LLMs struggle with historical analogy retrieval due to a focus on surface features, but the new CANA framework boosts their performance by 10% through causal understanding.
Systematic comparisons between current situations and structurally similar past events in the historical, i.e., historical analogies, is among the most powerful tools for foresight analysis. In this work, we present a new task called Analogical Deep Research (ADR) to Large Language Model (LLM) agents and construct the first ADR benchmark ADR-bench to study whether LLM agents are able to find and leverage historical analogies when doing foresight analysis. Our investigation reveals a key obstacle: LLM agents are poor at finding analogies because they match on surface features rather than underlying mechanisms. We argue that ADR is inherently a causal question as it requires understanding why the event occurred. Based on our theoretical analysis, we propose two principles required for ADR, including the mechanism alignment and cross-analogy confirmation. Built upon our theoretical results, we propose a new agentic framework called Causal Analogical Researcher (CANA) that guides LLMs to find and integrate historical analogies. CANA incorporates a simple yet effective structural decomposition representation, and integrates structural feedback for reflective improvements of historical analogy identification and integration. We show that CANA brings up to 10% improvements in historical analogy generation, and surpasses the state-of-the-art deep research agents in the ADR-bench. Case studies with the ongoing events confirm the effectiveness of CANA in leveraging historical analogies.
CMU ML