Daniel E. Ho

A framework for evaluating vulnerability reports via probes, executable checks of security properties, which surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.

Standard AI benchmarks often measure evaluation formatting rather than their purported constructs—with "reasoning" and "knowledge" metrics failing to discriminate from one another, and bias benchmarks correlating more strongly with general reasoning than with peer bias tests.

Switching an LLM from an API to a consumer chat interface can degrade performance more than downgrading an entire model generation—and standard API hyperparameter controls cannot reliably bridge the gap.

Matched comparisons reveal that LLMs may reproduce training sequences at alarming rates, but many of these instances are actually false positives rather than true memorization.