Search papers, labs, and topics across Lattice.
The paper introduces HindSight, a time-split evaluation framework for AI-generated research ideas that assesses quality by matching them against future publications and scoring them based on citation impact and venue acceptance. This framework restricts idea generation systems to pre-cutoff literature and evaluates outputs against papers published in the subsequent 30 months. Experiments across 10 AI/ML research topics demonstrate that HindSight reveals a significant performance difference between retrieval-augmented and vanilla idea generation, a difference not captured by LLM-as-Judge evaluations, and further shows a negative correlation between HindSight scores and LLM-judged novelty.
LLM-as-Judge evaluations of AI-generated research ideas can be misleading, as they often fail to correlate with real-world impact, even overvaluing ideas that never materialize in actual research.
Evaluating AI-generated research ideas typically relies on LLM judges or human panels -- both subjective and disconnected from actual research impact. We introduce \hs{}, a time-split evaluation framework that measures idea quality by matching generated ideas against real future publications and scoring them by citation impact and venue acceptance. Using a temporal cutoff~$T$, we restrict an idea generation system to pre-$T$ literature, then evaluate its outputs against papers published in the subsequent 30 months. Experiments across 10 AI/ML research topics reveal a striking disconnect: LLM-as-Judge finds no significant difference between retrieval-augmented and vanilla idea generation ($p{=}0.584$), while \hs{} shows the retrieval-augmented system produces 2.5$\times$ higher-scoring ideas ($p{<}0.001$). Moreover, \hs{} scores are \emph{negatively} correlated with LLM-judged novelty ($蟻{=}{-}0.29$, $p{<}0.01$), suggesting that LLMs systematically overvalue novel-sounding ideas that never materialize in real research.
