MIT CSAIL

Deployment rules can shift multi-agent AI outcomes dramatically, with fatality rates varying by up to 58 percentage points based solely on the chosen rule.

Most enzyme specificity models fail to outperform basic sequence alignment methods, highlighting a critical gap in predictive capabilities.

User ratings of LLMs are more about what users expect than the actual performance, revealing a critical flaw in how we assess AI models.

The choice of performance metrics could determine whether AI capabilities remain concentrated among the wealthy or proliferate across a broader developer base.

Evaluating creative AI requires recognizing that professional disagreement reflects genuine taste differences, not just noise in measurement.

Memory consistency in video generation models falters significantly when objects disappear, with state-of-the-art models struggling to recover updated states upon reappearance.

Current evaluation metrics for AI-powered AAC systems overlook the intersectional nuances of user needs, risking ineffective communication solutions.

LLMs show significant variability in the actionability of their UX critiques, with some models outperforming others across different product categories.

LLMs miss over 50% of errors in human-written code, but with test-time scaling, they can identify issues in more than 90% of cases—if you can afford the compute.

Current AI agents excel in structured tasks but falter at generating novel insights and tackling open-ended scientific challenges.

Even state-of-the-art multimodal models struggle with reliability in clinical tool use, revealing critical gaps in AI agent performance.

Optimizing input configurations can boost LLM performance in pathology tasks, closing the gap with specialized models and challenging assumptions about domain-specific training.

Systematic gaps in AI evaluation reporting are exposed, revealing inconsistencies that hinder reliable comparisons across thousands of models and benchmarks.

SeClaw reveals that existing benchmarks fall short in capturing the complexities of agent behavior, enabling a more nuanced evaluation of security risks in autonomous systems.

Multimodal pretraining doesn't guarantee better alignment with human reading patterns, suggesting that language-internal representations are still king when modeling how humans process text.

You can now audit Rényi differential privacy with near-optimal sample complexity, thanks to a new framework that directly estimates Rényi divergence using Donsker-Varadhan estimators.

Coordinating AI agents across scientific disciplines only boosts performance when each discipline captures a unique piece of the puzzle, otherwise, simpler combined summaries often suffice.

LMs encode grammaticality as a distinct feature in their hidden representations, separable from raw string probability and generalizable across languages.

Even the best LLMs still stumble on Olympiad-level math, and retrieval quality is the bottleneck for retrieval-augmented problem solving, according to the new MathNet benchmark.

LLMs play favorites: GPT-5-nano is significantly more likely to agree with incorrect statements depending on the perceived race, age, gender, and confidence of the user.

Stop wasting time wrestling incompatible transportation datasets: Ozone slashes experiment setup by 85% and boosts cross-city transfer of safety models by 91%.

Gaze-tracking unlocks a new level of personalized AI assistance, enabling LLMs to infer user cognitive states and boost recall performance.

Soft-gating with an "advisor" model can steer LLMs to be safer and more useful, reducing over-refusal without sacrificing detection accuracy.

LLM agent skills, despite their promise, often fail in realistic settings, with performance plummeting to no-skill baselines when agents must retrieve skills from a large, uncurated collection.

Particle filter models of sentence processing inherently predict "digging-in" effects—where disambiguation difficulty increases with the length of the ambiguous region—a phenomenon not captured by surprisal-based models.

Fine-tuning unlocks LLMs' surprising ability to predict how memorable a sentence is and how long it takes to read, exceeding traditional methods.

Building a complete web application from scratch remains a surprisingly hard task for even the best AI models, with top performance at only 58% accuracy on a new end-to-end benchmark.

Forget simulated manipulation—ManipulationNet offers a global infrastructure for benchmarking robots in the real world, complete with standardized hardware and software, to finally measure progress toward general manipulation.

VLMs are nowhere near human-level general intelligence: they score less than 10% of human performance across a diverse set of human-designed games, especially struggling with world-model learning, memory, and planning.

HybridRAG-Bench reveals that existing benchmarks overestimate the reasoning abilities of retrieval-augmented LLMs due to contamination, offering a more realistic evaluation using up-to-date scientific knowledge.