Search papers, labs, and topics across Lattice.
This paper introduces PerceptionRubrics, a novel evaluation framework designed to bridge the gap between high benchmark scores and real-world performance by employing a rigorous rubric-based approach. By pairing 1,038 information-dense images with over 10,000 instance-specific rubrics derived from a Circular Peer-Review consensus, the framework emphasizes essential facts and fine-grained details through a dual-stream system. The findings reveal critical insights into model brittleness, including a significant reliability gap and an 8% perception deficit between open-source and proprietary models, underscoring the importance of strict perceptual fidelity for reliable AI generation.
Models may score well on benchmarks but often fail to meet strict perceptual requirements, revealing a hidden brittleness in multimodal evaluations.
We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 10,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.
