Search papers, labs, and topics across Lattice.
The paper introduces Scalable Delphi, a method leveraging Large Language Models (LLMs) to emulate the Delphi method for structured expert elicitation in quantitative risk assessment. It addresses the limitations of traditional Delphi methods, which are time-consuming and resource-intensive, by using LLMs with diverse personas, iterative refinement, and rationale sharing. The study demonstrates that LLM panels can achieve strong correlations with benchmark ground truth, improve with added evidence, and align with human expert panels, suggesting LLMs can serve as scalable proxies for expert elicitation.
LLMs can now perform structured expert risk elicitation in minutes, rivaling human expert panels and slashing the months-long timelines of traditional Delphi methods.
Quantitative risk assessment in high-stakes domains relies on structured expert elicitation to estimate unobservable properties. The gold standard - the Delphi method - produces calibrated, auditable judgments but requires months of coordination and specialist time, placing rigorous risk assessment out of reach for most applications. We investigate whether Large Language Models (LLMs) can serve as scalable proxies for structured expert elicitation. We propose Scalable Delphi, adapting the classical protocol for LLMs with diverse expert personas, iterative refinement, and rationale sharing. Because target quantities are typically unobservable, we develop an evaluation framework based on necessary conditions: calibration against verifiable proxies, sensitivity to evidence, and alignment with human expert judgment. We evaluate in the domain of AI-augmented cybersecurity risk, using three capability benchmarks and independent human elicitation studies. LLM panels achieve strong correlations with benchmark ground truth (Pearson r=0.87-0.95), improve systematically as evidence is added, and align with human expert panels - in one comparison, closer to a human panel than the two human panels are to each other. This demonstrates that LLM-based elicitation can extend structured expert judgment to settings where traditional methods are infeasible, reducing elicitation time from months to minutes.
