Search papers, labs, and topics across Lattice.
This study explores the application of an LLM, specifically Claude from Anthropic, to solve the first thirty-three problems from the Ninety-Nine Prolog Problems (P-99) set by generating Prolog code and accompanying test files. The approach combines vibe-coding, where informal English specifications guide the code generation, with vericoding, as the generated code undergoes formal verification using the Logic Program Theorem Prover (LPTP) to ensure properties like termination and functional correctness. The results include the generation of 58 logic procedures, 508 tests, and 257 lemmas, culminating in 11,800 proof lines, demonstrating the potential of LLMs in automating both coding and formal verification tasks.
Claude generated 58 logic procedures and proved their correctness, showcasing the power of LLMs in automating complex programming tasks with formal guarantees.
Ninety-Nine Prolog Problems (P-99) is a famous set of Prolog exercises. We solved the first thirty three just by prompting an LLM (Large Language Model). We used Claude from Anthropic. By solved we mean: generate the Prolog code and a test file, run the tests and check whether they pass, then formally prove types, groundness, termination, uniqueness, existence and also sometimes functional correctness with LPTP (Logic Program Theorem Prover). Hence our approach is an experiment in vibe-coding/vericoding of P-99. It is a vibe-coding experiment because we started from informal specifications written in English and let Claude generate the Prolog code. It also fits within vericoding because the LLM proved reliability guarantees on the generated Prolog code. Claude wrote 58 logic procedures, 508 tests, 257 lemmas for a total of 11800 proof lines. We manually checked each file generated by the LLM. We checked the Prolog code, ran the tests, examined the logical statements generated by Claude and proof-checked Claude's proofs with LPTP. This paper describes this experiment and provides the main details so that it can be reproduced by the interested reader.
