Search papers, labs, and topics across Lattice.
VeriHarness scales agentic verification for long-horizon LLM tasks by leveraging repeated sampling to identify complementary correct claims and using the LLM itself as an agentic verifier with a workspace and evidence tools. The system employs a disagreement resolver to check competing claims against environmental evidence and a consensus challenger to test shared claims and identify omitted requirements, leading to improved selection and revision of final artifacts. Across five benchmarks and two frontier models, VeriHarness significantly outperforms baselines, with evidence-backed revision yielding substantial gains over single rollouts.
LLM agents can be turned into powerful verifiers by giving them a workspace and evidence tools, outperforming baselines on complex long-horizon tasks by resolving disagreements and challenging consensus.
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.
Google Research