Search papers, labs, and topics across Lattice.
The REAP system was developed for the AKBC Shared Task 2026 to construct knowledge bases from language models without fine-tuning, utilizing a budget of 32B parameters. By integrating structured chain-of-thought reasoning, relation-specific query strategies, and a reasoning-based empty-set gate, REAP effectively elicits and extracts knowledge into valid JSON arrays. The system achieved a macro-F1 score of 0.62 on the test set, with exceptional performance in specific relation categories, highlighting its potential for efficient knowledge base construction from LLMs.
REAP achieves a remarkable macro-F1 score of 0.62 in closed-book knowledge base construction, showcasing the power of structured reasoning without model fine-tuning.
We present the REAP system for the AKBC Shared Task 2026 on constructing knowledge bases from language models in a closed-book setting, subject to a budget of at most 32B parameters and no model fine-tuning. Our system combines structured chain-of-thought reasoning, relation-specific query strategies, and a reasoning-based empty-set gate to elicit parametric knowledge, followed by direct extraction into valid JSON arrays. On the test set, the system, built on the Mistral-Small-24B-Instruct-2501 model, achieves a macro-F1 score of 0.62, with particularly strong results on countryLandBordersCountry (F1 = 0.95), companyTradesAtStockExchange (F1 = 0.73), and hasArea (F1 = 0.77). Our code is publicly available at https://github.com/yammdd/AKBC-Shared-Task-2026.
