Search papers, labs, and topics across Lattice.
This study investigates the ability of large language models (LLMs) to autonomously generate complete and correct Answer Set Programming (ASP) theories within a one-hour time limit, utilizing a neurosymbolic approach with a solver integrated into the process. The authors evaluated nine models across three challenging visual question answering (VQA) benchmarks, revealing that several frontier models achieved high accuracy, with three models scoring 100% on CLEVR and near-perfect results on GQA and CLEVRER. Notably, while most models performed well, GPT-5 exhibited significant variability, highlighting the impact of model architecture on task performance and the potential for fine-tuning with reference theories.
Three frontier models achieved near-perfect accuracy in generating ASP theories, while GPT-5's performance varied dramatically, revealing the critical influence of model design on task success.
Writing Answer Set Programming (ASP) theories from scratch is a difficult and time-consuming task. We take a neurosymbolic approach to study whether a model can distill complete and correct theories, given a fixed agent harness with the solver in the loop. The protocol is dataset-agnostic: with a single prompt and an empty file as the starting point the model is given a 1-hour time limit to derive a complete theory. We chose VQA as the application domain, three benchmarks (CLEVR, GQA, CLEVRER), as these are publicly available and non-trivial. In order to study the model scale required for solving this task we nine different models: four frontier (Claude Sonnet 4.6, Claude Opus 4.7, GPT-5, DeepSeek V4 Pro), two mid-tier (DeepSeek V4 Flash, gpt-oss-120b), and three open-weights (qwen3.6-27b, gpt-oss-20b, qwen3.5-9b). Three of four frontier models reach 100% on CLEVR and 92.8%-98.8% on GQA; on CLEVRER, Sonnet, Opus, DeepSeek V4 Pro score 92.7%-95.3%. GPT-5 reaches 98.7% on CLEVR but drops to 41.8% on GQA and to 86.7% on CLEVRER. Adding handwritten reference theories from other datasets moves the other three frontier models by at most +/-3.4 pp but reduces GPT-5's accuracy by 3-19 pp. We release the code, prompts, and theories distilled.
