Microsoft Research

AI struggles with scientific lineage reasoning, with top models achieving only 27.3% accuracy, exposing critical gaps in our understanding of idea evolution.

VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.

Coding agents can generate observability artifacts, but they miss key diagnostic semantics, exposing fault signals for only 13.99% of failures.

Despite high diagnostic accuracy, LLMs fail to choose valid recovery actions for over 60% of incidents, exposing a critical flaw in their operational utility.

RustMizan exposes critical weaknesses in vulnerability detection, revealing that even advanced models struggle with line localization despite decent binary classification performance.

Code LLMs can recognize incorrect instructions but still follow them, leading to irrecoverable semantic errors that defy traditional evaluation metrics.

Despite the promise of advanced AI agents, even the best performers struggle, with Codex GPT-5.5 only achieving a 42% success rate in complex healthcare tasks.

Clarifying memories can significantly boost the factual accuracy and personalization of conversational agents, while irrelevant memories lead to degraded responses.

ConflictScore reveals that language models often overlook conflicting evidence, leading to overconfident and inaccurate claims.

Just two factors can explain over 90% of a model's performance across 133 benchmarks, drastically simplifying evaluation processes.

Evaluation awareness in language models reveals a significant gap between benchmark performance and real-world safety, challenging the reliability of current evaluation metrics.

Prospective memory in LLMs is not just harder than retrospective memory; it reveals critical insights into a model's reasoning capacity and attentional robustness.

High-diversity training improves safety in VLA models, but sub-optimal trajectory synthesis still hinders task success.

Simple prompting techniques can transform LLMs into more reliable mirrors of human judgment, recovering the full spectrum of responses.

Current AI agents excel in structured tasks but falter at generating novel insights and tackling open-ended scientific challenges.

Current models struggle with hybrid interface tasks, achieving only a 41.2% success rate, underscoring a critical gap in CUA evaluation.

Selective context pruning combined with automated summarization can boost LLM performance to over 91% in complex enterprise tasks while slashing token usage and processing time.

SeClaw reveals that existing benchmarks fall short in capturing the complexities of agent behavior, enabling a more nuanced evaluation of security risks in autonomous systems.

LLMs can nail trivia in English, but stumble in Indian languages – unless you throw in some code-mixing, which magically bridges the gap.

Multilingual LLM performance disparities aren't random noise: language features and model biases systematically explain up to 92% of the variance, revealing concrete targets for improvement.

Unlock zero-shot jailbreak defense: a model's vulnerability can be predicted and patched by understanding its "behavioral geometry" relative to other models, slashing probe costs by 98%.

Exposing full reasoning traces from LLMs can actually *hurt* user performance on reasoning tasks, suggesting they're more of a distracting interface element than a helpful window into the model's thought process.

AI can help structure fuzzy concepts like "fairness" and "reasoning" into measurable specifications, potentially streamlining the evaluation of GenAI systems.

Model-generated skills can actually hurt agent performance, and bigger models don't necessarily make for better skill extractors or consumers.

Knowing which component to tweak is half the battle: directly evaluating harness optimizers via priority ranking reveals whether they're making informed decisions or just stumbling upon improvements.

Current LLM jailbreak evaluations are inadequate, often relying on narrow metrics, necessitating a multi-dimensional framework like Security Cube for comprehensive security assessment.

Continuous benchmarking of protein function prediction models is now possible, enabling faster iteration and more robust performance tracking as annotations evolve.

Token-level attribution struggles to pinpoint the causes of LLM failures in realistic settings, suggesting current interpretability tools may not be up to the task of debugging complex model behaviors.

Despite impressive unit test pass rates, today's best LLMs rewrite code instead of precisely debugging it, achieving less than 45% edit precision even when explicitly instructed to minimize changes.

LLMs are twice as likely as humans to repeat the same support tactic in a conversation, but a simple RL reward for tactic novelty can fix it.

Gaze-tracking unlocks a new level of personalized AI assistance, enabling LLMs to infer user cognitive states and boost recall performance.

Knowing the *perfect* API to use or *exact* location to edit could drastically improve SWE agent performance, but knowing the perfect regression test result? Not so much.

Process-level evaluation reveals hidden skill discrepancies among web agents, enabling targeted improvements that traditional success metrics overlook.

People aren't as bothered by AI failing at easy tasks as you might think, suggesting our expectations for AI competence are more nuanced than a simple aversion to errors.

LLMs still fail to grasp research-level mathematics, with top models scoring below random chance when superficial pattern matching is removed, even with access to proof sketches.

AI-generated code's fluency masks a critical flaw: it often fails to deliver what users actually intend, highlighting the urgent need for "intent formalization" to bridge the gap between informal requirements and precise program behavior.

LLMs, even when prompted or fine-tuned, struggle to replicate the messy reality of human conversation, raising serious questions about their utility as proxies for social interaction.

LLMs' ability to fairly represent English dialects hinges on the quality of human consensus, revealing a fundamental challenge in improving performance for low-resource locales.

LLMs still can't automate real-world threat research, struggling with accuracy and nuanced expertise in a new benchmark derived from a world-leading company's CTI workflow.

LLMs writing long stories frequently contradict themselves on basic facts and timelines, especially in the middle of the narrative, highlighting a critical weakness in long-form generation.

LLMs can mimic your style, but your friends can still tell it's not really you, especially when it comes to your opinions.

LLMs can't reliably debug code in long contexts (64k-128k tokens) even with perfect information retrieval, despite impressive performance in agentic workflows that decompose the task.

LLM development teams often resort to workarounds and augmentation strategies when faced with the practical challenges of integrating domain experts, revealing a gap between ideal participatory design and real-world constraints.