Search papers, labs, and topics across Lattice.
This paper evaluates the efficacy of LLMs for grant proposal review using a perturbation-based framework across six quality axes on EPSRC proposals. They compare single-pass, section-by-section, and "Council of Personas" review architectures, finding that section-level analysis yields the best detection rate and scoring reliability. The study reveals that LLMs are better at identifying alignment issues than clarity flaws and tend to focus on compliance over holistic assessment.
LLMs excel at compliance-focused grant review but struggle with holistic assessment, revealing a critical gap in their ability to replace human expert panels.
As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap''for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a'Council of Personas'ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.
