Search papers, labs, and topics across Lattice.
This paper surveys approximately 150 verifiers for robot policies, categorizing them into four families based on their judgment sources: human, rule-based, learned, and model-intrinsic verifiers. The study reveals a trade-off between the availability and credibility of verifiers, where increased availability often leads to diminished credibility, highlighting the inherent limitations of relying on any single verifier type. The authors propose nine metrics to evaluate verifiers and discuss the implications for future development in this field, emphasizing that there is no "free checker" for robot policy evaluation.
Increased availability of verifiers often comes at the cost of their credibility, revealing a critical trade-off in robot policy evaluation.
A verifier for robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate vision-language-action policies and to train them. Verifiers range from success detectors and reward models to runtime monitors, safety filters, and temporal-logic specifications. We survey roughly 150 verifiers and compare them along two properties. Availability is how much a verdict costs, how early in a rollout the verdict arrives, and how often a verdict can be asked for. Availability rises as verdicts get cheaper, earlier, and denser. Credibility is how much a high score tells us about the task. Credibility falls as the judgment becomes gameable and self-serving. We group the verifiers by who supplies the judgment: human verifiers, rule-based and formal verifiers, learned and pretrained verifiers, and model-intrinsic verifiers. Across the four families, we find that credibility falls as availability rises. Regardless of who supplies the judgment, there is no free checker. We then examine what validates a verifier itself, and how much a high score tells us. Three measures appear in the literature: agreement with human labels, the performance of the policy it trains, and behavior under reward hacking. We close with nine metrics that make a verifier claim checkable, and coordinates for the verifiers still to be built.
