Anany Kotawala

Even when individual LLM agents are locally coherent, composing their probabilistic claims can lead to surprisingly large violations of basic probability axioms, causing significant regret in betting scenarios.

Many LLM leaderboard rankings are statistically meaningless: up to 27.5% of pairwise comparisons are unresolved even with standard statistical power.