Search papers, labs, and topics across Lattice.
This study evaluates the divergence between research ideas generated by large language models (LLMs) and those produced by human researchers, employing a novel large-scale framework. By reverse-engineering related prior works from high-quality human papers, the authors prompt LLMs to generate new ideas and analyze them using a two-axis research-taste taxonomy. The findings reveal that while LLMs can generate reasonable ideas, they tend to cluster around specific opportunity patterns, indicating a narrower scope compared to the broader and more diverse human-generated ideas.
LLM-generated research ideas are systematically narrower and more focused than those of human researchers, revealing a significant gap in creative breadth.
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.
