Search papers, labs, and topics across Lattice.
This study introduces VeriHarness, a structured feedback mechanism that enhances the performance of LLM agents by providing detailed failure diagnostics during the candidate generation process. By comparing raw diagnostics to structured feedback that includes failure location, observed value, and admissible alternatives, the authors demonstrate a significant increase in terminal success rates across multiple TextWorld games. Notably, the inclusion of admissible alternatives in feedback is identified as the key factor driving improvement, with success rates rising from 14 to 36 for Qwen2.5-Coder-14B and from 8 to 29 for Llama-3.1-8B under a four-call limit.
Structured feedback can boost LLM agent success rates by up to 44 percentage points, revealing the critical role of admissible alternatives in the repair process.
LLM agents often retry after external validation rejects a candidate, but the interface between validation and the next model call remains underspecified. We introduce VeriHarness, a code-controlled agent loop in which models generate candidates while external validators control acceptance, budgets, and traces. We use it to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games under a four-call cap, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 points). Ablations locate most of the gain in the admissible alternatives: feedback containing only the location and observed value remains near the raw diagnostic baseline. Presenting the complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. The ordering persists across the tested call budgets and one sampled-decoding setting.
NVIDIA