CMU Machine Learning

AI struggles with scientific lineage reasoning, with top models achieving only 27.3% accuracy, exposing critical gaps in our understanding of idea evolution.

Operational reframing emerges as a critical risk signal, revealing that compliance can vary significantly across models and scenarios, challenging the notion of stable safety metrics in multi-agent LLMs.

None of the 30 LLM agents evaluated in CausalGame demonstrated reliable causal thinking, revealing a critical gap in AI's ability to perform scientific reasoning.

Many robotic policies that seem successful in manipulation tasks actually compromise safety, with SoftVTBench revealing a stark contrast between goal completion and physical safety metrics.

PACE-Bench predicts agentic performance with remarkable accuracy while slashing evaluation costs to a fraction of traditional methods.

Traditional metrics fail to capture the true memory capabilities of LLMs, exposing a critical gap in how we assess their deployment readiness.

Even top-performing AI models struggle with PowerPoint tasks, achieving only 45% success rates despite a robust evaluation framework that rewards nuanced performance.

Current multimodal conversational models miss critical emotional cues, revealing a significant gap in their ability to engage in nuanced human-like dialogue.

Local names boost retrieval accuracy, but models still fail to generate images that faithfully represent specific street segments.

Training performance can significantly forecast real-life tutoring effectiveness, with open responses proving to be a stronger predictor than traditional assessments.

Traditional success metrics tie agent performance 75% of the time—this new approach slashes that to 35%, revealing clearer distinctions in agent capabilities.

LLMs can outperform humans in predicting the next speaker in meetings, even without audio or visual data.

LLM agents can identify reproducibility problems in 90% of analyzed machine learning papers, leveraging GitHub issues as a novel supervision source.

LLMs show significant variability in the actionability of their UX critiques, with some models outperforming others across different product categories.

Claude Opus 4.6 outperforms its peers, solving over half of the complex tasks in a personalized desktop environment, revealing critical gaps in current AI capabilities.

AIChilles reveals 49 hidden weaknesses in AI-evolved systems, challenging the assumption that AI-generated code is always superior to human-designed algorithms.

High-performance ML models can be reproduced with minimal information, revealing that they thrive in low-complexity regions and defy traditional overfitting concerns.

AI coding agents can reliably reproduce social science findings, with Claude Code significantly outperforming Codex in this capacity.

A novel hacker-fixer loop can eliminate reward hacking vulnerabilities in agent benchmarks, transforming how we secure AI evaluation metrics.

iOSWorld reveals that even state-of-the-art models falter in multi-app reasoning, achieving only 37% accuracy, underscoring the complexity of personal context in AI interactions.

LLMs fail to deliver personalized responses that align with human judgments, often producing results indistinguishable from generic outputs.

Leading LLMs falter in Korean web-browsing tasks, achieving less than half the accuracy found in previous benchmarks.

Even the top-performing LLM struggles with realistic user interactions, achieving only 61% success in complex task scenarios.

Over 54% of actions taken by leading LLM coding agents in realistic projects result in harmful safety violations, exposing critical gaps in current safety alignment.

LLMs can ace prediction tasks in causal environments, but still fail to grasp the underlying causal mechanisms, revealing a critical blind spot in their reasoning abilities.

Healthcare LLM benchmarks can be misleading because they fail to capture critical assumptions about how users interact with models and how those interactions translate to real-world outcomes.

Stop blindly trusting synthetic data for agent evaluation: SynAE reveals that no single metric can fully capture its quality, demanding a multi-faceted approach.

LLMs may only account for 11-26% of high-level goal-setting in collaborations, but they exert far more influence by shaping the micro-decisions and concrete requirements that define those goals.

Existing robotic methods falter in tackling fundamental physical reasoning challenges, as evidenced by KinDER's rigorous benchmark evaluation.

Today's best web agents are shockingly inefficient, achieving only 1.15% trajectory efficiency on realistic long-horizon tasks, revealing a critical need to move beyond simple success rates.

Continual learning for LLM agents hits a wall: scaling models doesn't reliably improve skill generation, and self-feedback can lead to recursive drift.

Current user modeling benchmarks are child's play compared to the real-world challenges exposed by HORIZON, a massive new dataset spanning 54M users and diverse domains.

Frontier LLMs are surprisingly vulnerable to a wide range of task-specific exploits, from simple output spoofing to rootkit-style binary hijacking, even in seemingly well-defined environments.

Forget carrots and sticks: contracts and mediation are the surprisingly effective keys to unlocking cooperation between LLMs, even when individual incentives push toward defection.

Agentic coding gets a serious boost: distilling and reusing rollout trajectories lets Claude-4.5-Opus jump from 70.9% to 77.6% on SWE-Bench Verified.

Stop evaluating AI systems in isolation: marketplace dynamics like user switching and early-adoption advantages critically shape real-world success.

LLMs can mimic human writing, but not as well as you think: genre matters more than the source (human vs. LLM), and model choice trumps decoding strategy when it comes to style.

Data augmentation with LLMs can tank your NER performance even when it boosts POS tagging, proving task structure matters more than synthetic data quality.

DPO might not be the only game in town: a decision-directed approach to reward modeling can outperform it in pairwise preference optimization.

Interpretability methods often fail to improve over black-box prompting when models are uncooperative, suggesting current techniques may be more about elicitation than revealing internal mechanisms.

SAM models exhibit surprisingly divergent behaviors under occlusion, with some prioritizing visible tissue and others confidently hallucinating hidden anatomy.

Stop reimplementing multimodal models: TorchUMM offers a unified codebase for evaluation, analysis, and post-training, streamlining research across diverse architectures and tasks.

Today's best AI agents can only complete 33% of common online tasks like booking appointments or filling out job applications, revealing a significant gap between current capabilities and real-world utility.

LLMs struggle to synthesize scientific conclusions from structured biomedical evidence, and current metrics fail to capture nuanced differences in their reasoning abilities.

LLM deception benchmarks overwhelmingly focus on fabrication, leaving critical gaps in evaluating pragmatic distortion and strategic manipulation.

Just 10 minutes of AI assistance can measurably degrade your ability to solve problems on your own.

LLM-powered forums may generate norm-aware language, but they fail to foster the crucial back-and-forth needed for communities to teach, enforce, and revise those norms.

LLMs get *more* honest when they have time to reason, defying human tendencies and revealing surprising insights about their internal representational geometry.

Finally, a standardized benchmark for survival analysis HTE estimation lets you rigorously compare methods across synthetic, semi-synthetic, and real-world datasets.

Forget simulated manipulation—ManipulationNet offers a global infrastructure for benchmarking robots in the real world, complete with standardized hardware and software, to finally measure progress toward general manipulation.