Search papers, labs, and topics across Lattice.
This study reveals that traditional finite-reference-plus-matcher pipelines can mislabel unmatched outputs as false, leading to incorrect model selection in open-ended Theory-of-Mind (ToM) tracking. By recoding references, the authors demonstrate a significant reduction in weighted prevalence and a reversal in Brier risk, indicating that many unmatched beliefs are valid and should not be disregarded. Their findings highlight the importance of accurate calibration in belief tracking, showing that a majority of unmatched beliefs are indeed true, which has implications for the evaluation of AI systems' reasoning capabilities.
Unmatched beliefs in Theory-of-Mind tracking are often valid, and mislabeling them can lead to significant errors in model selection and calibration.
Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references. A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection on fixed outputs. Holding 259 beliefs and paired scores fixed, reference recoding lowers weighted prevalence from 0.783 to 0.295 and reverses strictly proper Brier risk: a frozen source-prior rule leads native confidence by 0.227 under reference labels and trails by 0.152 under blinded adjudication, in all six authored scenarios. A reference-only Platt recalibrator reverses further. An ICE-specific reversal appears in a released 301-question NQ-open DPR-BERT pipeline: its average-confidence baseline improves instance-level calibration error by 0.045 under exact match but worsens it by 0.074 under human correctness, with both intervals excluding zero. On independently authored OpenToM narratives, 90-96% of audited unmatched beliefs are literally true and the paired direction again reverses. An exact decomposition attributes the distortion to omitted truths, and a closed-form criterion correctly classifies comparisons from twelve released systems. Frozen-audit retrospective replay shows 50 attempted annotations recover ranking direction with probability at least 0.996. TriSource-Restore anchors full-frame reference labels and frozen automatic judgments to a probability-sampled human pilot, maintains at least nominal coverage, narrows intervals, and repairs confidence subject to a base-rate deployment gate.
