Search papers, labs, and topics across Lattice.
This paper introduces a novel paradigm for measuring intelligence that transcends human capability by leveraging relative measurement through model-generated public challenges. The authors highlight the limitations of traditional human-authored benchmarks and propose an adversarial psychometric rating system that scales with the capabilities of the systems being evaluated. Key findings demonstrate that this framework can effectively assess both verifiable and non-verifiable domains, enabling ongoing evaluation of AI systems as they surpass human performance thresholds.
Relative measurement through model-generated challenges could redefine how we evaluate intelligence beyond human limits.
How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.
