Search papers, labs, and topics across Lattice.
The paper investigates the impact of self-consistency decoding on recalling encyclopedic knowledge, addressing a gap in understanding its effectiveness beyond symbolic reasoning. They create a knowledge recall split for the MMLU benchmark, validated by performance patterns analogous to MedMCQA. The key finding is that self-consistency consistently improves performance on both symbolic reasoning and knowledge recall, leading to state-of-the-art 89% accuracy on MMLU using GPT-4o.
Self-consistency, previously known for boosting reasoning, turns out to be a surprisingly effective way to make LLMs remember facts too.
While self-consistency is known to improve performance on symbolic reasoning, its effect on the recall of encyclopedic knowledge is unclear due to a lack of targeted evaluation grounds. To address this, we establish such a knowledge recall split for the popular MMLU benchmark by applying a data-driven heuristic from prior work. We validate this split by showing that the performance patterns on the symbolic reasoning and knowledge recall subsets mirror those of GSM8K and MedMCQA, respectively. Using this solid ground, we find that self-consistency consistently improves performance across both symbolic reasoning and knowledge recall, even though its underlying CoT prompting is primarily effective for symbolic reasoning. As a result, we achieve an 89\% accuracy on MMLU, the best performance to date with the use of GPT-4o.
