Stanford HAI

Even the best AI agents fail to deliver substantial business value in realistic customer delivery scenarios, struggling to uncover unstated constraints and achieve high impact.

Frontier LLM agents fail to recognize superior human-corrected scientific research plans, performing worse than chance even with full project context.

Autonomous agents can now discover their own evaluation metrics, bypassing the costly bottleneck of real-world engineering benchmarks.

Future LLM agents can be prevented from reward hacking by using today's models to automatically harden their evaluation environments.

AI agents are now building climate models that outperform published research and offer novel hypotheses for long-standing scientific debates.

Even advanced LLMs struggle to replicate human researchers' ability to find inspiring prior work, highlighting a significant gap in AI's scientific intuition.

VideoGen-Agent is presented, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation to support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.

A framework for evaluating vulnerability reports via probes, executable checks of security properties, which surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.

This work introduces PosteriorBench, a benchmark for evaluating the distributional accuracy of generative inverse solvers, and reveals substantial distribution-matching gaps across current solvers, while showing that neural operators improve resolution robustness, and guidance weights and generation noise are key to posterior-variance calibration.

This work instantiates Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies that meets the real-time control requirements of dynamic real-world manipulation, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics.

Upper-bounding the number of distribution shifts in sequential data is provably impossible without parametric assumptions, but conformal inference can uniquely deliver valid, non-trivial lower bounds.

Standard motion benchmarks fail inside the cabin because drivers mostly sit still—until right before a maneuver, when arm movement spikes 3.4x.

Switching an LLM from an API to a consumer chat interface can degrade performance more than downgrading an entire model generation—and standard API hyperparameter controls cannot reliably bridge the gap.

LLMs can autonomously generate prompts that rival expert-written ones, but still fall short in accurately interpreting scientific context and discovering literature.

Safety evaluations reveal that 6-21% of successful robotic manipulation rollouts still violate safety specifications, underscoring a critical gap in current methodologies.

Despite executing target instructions, LLMs often fail to deliver competitive performance in GPU kernel optimization, especially on complex tasks.

Skill-switching accuracy in LLMs drops significantly on complex tasks, but a new training approach boosts performance from 34.4% to 68.4% on challenging benchmarks.

Extending context in conversations can significantly amplify the risk of LLMs promoting delusional behaviors, challenging assumptions about model size and reasoning capabilities.

AI agents can handle the engineering aspects of research but fall short in addressing the core research questions, leading to outright rejections from experts.

Proprietary MLLMs may achieve high diagnostic accuracy, but they still struggle with reliable clinical reasoning, revealing significant gaps in their practical utility.

Matched comparisons reveal that LLMs may reproduce training sequences at alarming rates, but many of these instances are actually false positives rather than true memorization.

VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.

Algorithms with formal guarantees can effectively unlearn data, while many popular empirical methods fail dramatically, revealing a critical gap in current practices.

Unlearning shortcuts doesn't guarantee their complete removal; ART reveals that some associations can still be functionally restored, challenging existing evaluation methods.

Continuous scoring from LLM-as-a-Verifier leads to state-of-the-art verification accuracy and improved sample efficiency in reinforcement learning tasks.

EQMs reveal that explanation quality can be quantitatively assessed, offering a more reliable indicator of forecasting accuracy than traditional methods.

Terminal-use agents are still far from achieving reliable general-purpose performance, with top models only scoring 65.8% on a new benchmark that spans diverse real-world tasks.

None of the 18 multimodal large language models audited are order-invariant, with flip rates revealing a staggering sensitivity to input ordering that challenges current evaluation practices.

Fine-tuned behavioral models can achieve superior population-level alignment, closing the gap with general-purpose models in individual predictions.

Current AI models miss critical tumor detections in underrepresented demographics, revealing a hidden bias that could compromise patient outcomes.

ChartWalker reveals significant performance gaps in cross-chart RAG tasks, challenging the status quo of existing benchmarks and paving the way for more robust multi-modal reasoning.

No automatic metric can effectively balance validity and discriminative power in evaluating LLM-generated responses, revealing a fundamental limitation in current evaluation practices.

No single model dominates video embedding tasks, revealing stark contrasts in performance based on modality and task type.

High artifact detection rates in VLMs mask significant failures in contextual understanding, with top models misidentifying visual cues in over 46% of cases.

Current AI agents excel in structured tasks but falter at generating novel insights and tackling open-ended scientific challenges.

Systematic gaps in AI evaluation reporting are exposed, revealing inconsistencies that hinder reliable comparisons across thousands of models and benchmarks.

Behavioral safety metrics can mask significant latent vulnerabilities, with dissociated models revealing a stark contrast between outward behavior and internal robustness.

Current unlearning methods can ace the test but still flunk causal reasoning, and this paper introduces a benchmark and method to fix that.

VideoLLMs are surprisingly bad at keeping track of who did what, frequently mixing up actions across different video segments like a confused movie editor.

Over a quarter of tasks in popular AI benchmarks contain critical flaws that distort model evaluations, and this automated auditing framework can catch them.

Rubric-based scoring, a common evaluation method, is surprisingly bad at capturing quality in expert domains, with pairwise comparisons proving far more reliable and efficient.

Current video generation benchmarks miss the forest for the trees: EvalVerse actually measures cinematic quality, not just prompt adherence.

People systematically overestimate the efficiency gains from using AI for simple tasks, even when it wastes their time.

Despite impressive headline accuracy, today's AI chatbots exhibit alarming regional biases, near-total dependence on retrieval quality, and surprising vulnerability to subtle falsehoods in user queries when used as news intermediaries.

Despite their increasing role in scientific discovery, today's AI models are surprisingly bad at predicting which scientific breakthroughs will actually happen and when.

AI agents are shockingly easy to manipulate into leaking API keys, deleting user data, and initiating unauthorized transactions across a wide range of real-world applications.

Current LLM agents are woefully inadequate for real-world clinical tasks, achieving only 46% success on a new benchmark that demands long-horizon reasoning and verifiable execution within electronic health records.

Model rankings on standard benchmarks can flip entirely when you optimize prompts for each LLM, so your "best" model might actually be the worst.

Forget chasing the biggest LLM – this benchmark reveals that smaller models (<2B params) can deliver 3x better energy efficiency and faster ROI in real-world industry deployments.