Daniel E. Ho

Standard AI benchmarks often measure evaluation formatting rather than their purported constructs—with "reasoning" and "knowledge" metrics failing to discriminate from one another, and bias benchmarks correlating more strongly with general reasoning than with peer bias tests.

Switching an LLM from an API to a consumer chat interface can degrade performance more than downgrading an entire model generation—and standard API hyperparameter controls cannot reliably bridge the gap.