Mila

Even advanced LLMs struggle to prevent privacy breaches in multi-user settings, exposing critical data spillage risks that current benchmarks overlook.

Control interventions are often detected by LLMs, with awareness levels varying significantly across models and tasks, revealing vulnerabilities in AI safety protocols.

Systematic gaps in AI evaluation reporting are exposed, revealing inconsistencies that hinder reliable comparisons across thousands of models and benchmarks.

Code-switched speech can exploit safety weaknesses in LALMs, achieving jailbreak success rates that challenge current safety protocols.

Human-generated citation lists, long considered the gold standard for evaluating literature search, are surprisingly unreliable, with LLMs judging them relevant only ~50% of the time.

Even state-of-the-art vision-language models frequently lie and hallucinate when playing social deduction games, raising serious questions about their reliability in real-world applications requiring grounded reasoning.

Forget expensive downstream evaluations: token-level statistics from expert-written solutions can reliably forecast LLM performance with 10,000x less compute.

LLM-powered query reformulation, a hot topic in IR, often fails to translate gains from lexical to neural retrieval, and bigger models don't always help.

Forget hand-crafted prompts: RL can automatically unearth 36 new failure modes in VLMs that humans miss, revealing surprising blind spots in counting, spatial reasoning, and viewpoint understanding.

LLMs struggle to balance rational financial decisions with mimicking noisy user behavior, often overfitting to short-term market trends instead of aligning with long-term investment goals.