Search papers, labs, and topics across Lattice.
The paper introduces GPS-Bench, a benchmark designed to enhance the validity of LLM-based policy simulations by grounding them in real-world evidence from legislative records and other public documents. By reconstructing actors from historical data rather than using archetypal prompts, the benchmark facilitates a controlled comparison of various simulation methods, including joint reasoning and independent agent interactions. The findings reveal that fine-tuning on grounded records significantly improves actor-level impact predictions, while also providing insights into the mechanisms of coalition formation among actors.
Grounded in real-world evidence, GPS-Bench reveals that fine-tuning LLMs on historical policy data dramatically enhances the accuracy of multi-agent simulations in predicting policy outcomes.
Policy analysis requires more than predicting whether a proposal will pass: it requires identifying who will be affected, how those actors respond, and what follows. LLM-based policy simulations model these processes at scale, but their validity is hard to establish when plausible behaviour is never compared with observed outcomes. We introduce GPS-Bench, an evidence-grounded benchmark for governance policy simulation that links policies to relevant actors, actor actions and downstream impacts using legislative records, lobbying disclosures, regulatory documents, corporate filings, economic data and other public evidence. Actors are reconstructed from the dated record rather than prompted as archetypes, so a persona is an evidence object with provenance; a human-annotated pool forms the Gold evaluation set, while cases labelled by a separate LLM from retrieved evidence are treated as Silver supervision and never as test labels. Because every inference mode reads the same grounded state and emits the same schema, GPS-Bench turns"does multi-agent simulation help?"into a controlled comparison: we contrast joint reasoning, independent and communicating actor agents, graph-based methods and weight-level fine-tuning over one policy state. Fine-tuning on the grounded record gives the strongest actor-level impact prediction, and decomposition does not beat it; what decomposition adds is mechanism. Agents hold private, non-identical evidence, each seeing its own exposure clause, and address named partners with concrete joint proposals, what they offer, what they need in return, and why acting together beats acting alone, so the coalitions that form can be checked against the commitments the record holds. GPS-Bench therefore gives a common empirical setting for studying when evidence, actor modelling and multi-agent interaction improve the prediction and interpretation of policy outcomes.
