Stanford HAI

VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.

Algorithms with formal guarantees can effectively unlearn data, while many popular empirical methods fail dramatically, revealing a critical gap in current practices.

Unlearning shortcuts doesn't guarantee their complete removal; ART reveals that some associations can still be functionally restored, challenging existing evaluation methods.

Continuous scoring from LLM-as-a-Verifier leads to state-of-the-art verification accuracy and improved sample efficiency in reinforcement learning tasks.

EQMs reveal that explanation quality can be quantitatively assessed, offering a more reliable indicator of forecasting accuracy than traditional methods.

Terminal-use agents are still far from achieving reliable general-purpose performance, with top models only scoring 65.8% on a new benchmark that spans diverse real-world tasks.

None of the 18 multimodal large language models audited are order-invariant, with flip rates revealing a staggering sensitivity to input ordering that challenges current evaluation practices.

Fine-tuned behavioral models can achieve superior population-level alignment, closing the gap with general-purpose models in individual predictions.

Current AI models miss critical tumor detections in underrepresented demographics, revealing a hidden bias that could compromise patient outcomes.

ChartWalker reveals significant performance gaps in cross-chart RAG tasks, challenging the status quo of existing benchmarks and paving the way for more robust multi-modal reasoning.

No automatic metric can effectively balance validity and discriminative power in evaluating LLM-generated responses, revealing a fundamental limitation in current evaluation practices.

No single model dominates video embedding tasks, revealing stark contrasts in performance based on modality and task type.

High artifact detection rates in VLMs mask significant failures in contextual understanding, with top models misidentifying visual cues in over 46% of cases.

Current AI agents excel in structured tasks but falter at generating novel insights and tackling open-ended scientific challenges.

Systematic gaps in AI evaluation reporting are exposed, revealing inconsistencies that hinder reliable comparisons across thousands of models and benchmarks.

Behavioral safety metrics can mask significant latent vulnerabilities, with dissociated models revealing a stark contrast between outward behavior and internal robustness.

Current unlearning methods can ace the test but still flunk causal reasoning, and this paper introduces a benchmark and method to fix that.

VideoLLMs are surprisingly bad at keeping track of who did what, frequently mixing up actions across different video segments like a confused movie editor.

Over a quarter of tasks in popular AI benchmarks contain critical flaws that distort model evaluations, and this automated auditing framework can catch them.

Rubric-based scoring, a common evaluation method, is surprisingly bad at capturing quality in expert domains, with pairwise comparisons proving far more reliable and efficient.

Current video generation benchmarks miss the forest for the trees: EvalVerse actually measures cinematic quality, not just prompt adherence.

People systematically overestimate the efficiency gains from using AI for simple tasks, even when it wastes their time.

Despite impressive headline accuracy, today's AI chatbots exhibit alarming regional biases, near-total dependence on retrieval quality, and surprising vulnerability to subtle falsehoods in user queries when used as news intermediaries.

Despite their increasing role in scientific discovery, today's AI models are surprisingly bad at predicting which scientific breakthroughs will actually happen and when.

AI agents are shockingly easy to manipulate into leaking API keys, deleting user data, and initiating unauthorized transactions across a wide range of real-world applications.

Current LLM agents are woefully inadequate for real-world clinical tasks, achieving only 46% success on a new benchmark that demands long-horizon reasoning and verifiable execution within electronic health records.

Model rankings on standard benchmarks can flip entirely when you optimize prompts for each LLM, so your "best" model might actually be the worst.

Forget chasing the biggest LLM – this benchmark reveals that smaller models (<2B params) can deliver 3x better energy efficiency and faster ROI in real-world industry deployments.

FUSE achieves verification quality on par with semi-supervised methods, all without needing any labeled data.

LLMs are significantly more likely to spread misinformation about countries with lower Human Development Index and in lower-resource languages, revealing a concerning bias in their outputs.

People aren't as bothered by AI failing at easy tasks as you might think, suggesting our expectations for AI competence are more nuanced than a simple aversion to errors.

LLM performance hinges on the code around the model, and Meta-Harness proves that automating the design of this "harness" can significantly boost results across diverse tasks.

Educators in Hawai'i envision AI auditing tools that trace the genealogy of knowledge, highlighting the need for community-centered approaches to address cultural misrepresentation in AI.

LLMs' chain-of-thought reasoning often falls apart due to factual incompleteness, with errors compounding across multiple hops, as revealed by a new multi-hop QA dataset.

AI agents that ace isolated coding tasks fall apart when faced with the messy reality of continuous software evolution, dropping from 80% to 38% success rates in a new benchmark.

Current Large Audio Language Models (LALMs) struggle with basic audio understanding tasks like noise localization and cross-lingual speech, with some performing worse than random chance, despite excelling at speech recognition.

AI can generate realistic legal questions, but current models still struggle with diversity and a tendency to agree too much, revealing critical gaps in their ability to simulate adversarial legal reasoning.

Guaranteeing reductions in harm from biased LLM judges is now possible, even when the biases are unknown or adversarially discovered.

Achieve 50% lower latency in Verilog code generation without sacrificing accuracy by adaptively escalating between LLMs based on diagnostic feedback and formal verification.

Forget expert surveys: GPT-4.1-nano can predict the difficulty of data visualization test questions with surprisingly high accuracy, especially when combining visual and textual cues.

Turns out, the best memory design for robotic manipulation depends heavily on the task, with no single architecture dominating across the board.

Ensembling LLMs for educational tasks can backfire, worsening misalignment with actual learning outcomes despite improved benchmark performance.

LLMs struggle to explore multiple valid reasoning paths, often committing to a single route and missing alternative solutions, especially in complex, multi-step logical problems.

LLMs may grasp the broad strokes of causal strategies, but struggle with the devilish details of research design, as revealed by a new benchmark separating causal identification from estimation.

LLM-generated data can provide statistically valid causal effect estimates in social science, but only if you calibrate the simulations with real human data.

Language model capabilities are surprisingly stable over time for most tasks, except for math reasoning, which continues to advance, offering a way to reliably translate compute budgets into performance expectations.

A fine-tuned open-source Mistral-7B model rivals GPT-4 Turbo in extracting clinical history elements from imaging orders, offering a cost-effective and accurate alternative for assessing clinical history completeness.