A. Feder Cooper

Standard AI benchmarks often measure evaluation formatting rather than their purported constructs—with "reasoning" and "knowledge" metrics failing to discriminate from one another, and bias benchmarks correlating more strongly with general reasoning than with peer bias tests.

Headline claims that fine-tuning effortlessly extracts copyrighted books collapse under scrutiny, driven by sub-standard match thresholds, prompt leakage, and zero negative controls.

Matched comparisons reveal that LLMs may reproduce training sequences at alarming rates, but many of these instances are actually false positives rather than true memorization.

Insurers could face unprecedented losses without a robust framework to manage the unique risks posed by the rapidly evolving AI agent economy.