Search papers, labs, and topics across Lattice.
This paper introduces MAS-HQ, a resource-aware evaluation protocol designed to benchmark the factuality of AI systems while accounting for the computational costs associated with achieving that factuality. By comparing systems in a competitive setting rather than in isolation, the study reveals that models with higher raw factuality scores can be less efficient in deployment, as seen in the contrasting Q-Scores of a brute-force agent versus a more resource-efficient alternative. The findings indicate that competitive evaluation encourages the development of policies that balance factuality with resource efficiency, providing a clearer picture of the true cost of deploying AI systems.
Static leaderboards can mislead deployment decisions, as a system with higher factuality may actually be less efficient when compute costs are factored in.
On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource-aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5 in simulated preview) whose raw factuality scores are already bunched near the ceiling. MAS-HQ provides a reproducible way to measure how much a factual answer costs.
