Search papers, labs, and topics across Lattice.
This paper introduces a gradual knowledge excavation framework to address the limitations of LLMs in open-domain implicit question answering, specifically their reliance on internal knowledge and limited reasoning comprehensiveness. The framework iteratively acquires external information and reasons based on historical knowledge by selecting actions like querying external sources or performing logical reasoning steps. Experiments on the StrategyQA dataset demonstrate that the proposed method achieves 78.17% accuracy, setting a new state-of-the-art for ~10B-scale LLMs while using significantly fewer parameters than competitors.
LLMs can achieve state-of-the-art results on complex reasoning tasks with far fewer parameters by iteratively excavating and reasoning over external knowledge.
Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA for ~10B-scale LLMs.
