Microsoft Research

Even state-of-the-art LMMs struggle with reasoning-informed visual editing, with the best model only achieving 56.6% accuracy on a new comprehensive benchmark.

Choice-based models like Jev can accurately identify graph adjacencies but fail to grasp broader structural correctness, even with joint graph-text inputs.

Most video generation models fail to render text accurately, with even top performers exhibiting a 25% word error rate on a new benchmark designed for this critical oversight.

RAFT (Retrieval-Augmented Framework for Troubleshooting Agents) is introduced, a stateful RAG framework that abstracts each closed historical case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state.

ProgramDistill is introduced, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications, and provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.

Registers are implemented as dedicated fixed-position tokens whose continuous hidden states are trained to carry reasoning progress across generation chunks, which are especially effective for bounded code generation, where correct programs usually span several chunks.

This work introduces MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots, which performs better on reasoning than on planning, and the best-performing model remains far below observed human performance.

Industry-standard GPU memory sanitizers harbor major blind spots across core memory spaces, finally quantifiable through 149 targeted CUDA stress tests.

Current vision-language models falter in 3D assembly reasoning, struggling with accuracy and complexity in real-world industrial scenarios.

Achieving a single successful task completion in stateful workflows doesn't guarantee reliability, with many agents failing to maintain consistent performance across multiple attempts.

Current video generation models struggle with visual reasoning, with the best achieving only 51% accuracy on a new benchmark designed to probe their capabilities.

KNOWSIM uncovers that LLM performance is not uniform and shifts dramatically based on user knowledge levels, challenging traditional evaluation methods.

HalluTracer transforms hallucination detection by revealing that aggregating truth signals across model layers can dramatically enhance accuracy.

Abstaining from uncertain predictions can enhance LLM accuracy in medical applications by 9.6 percentage points, transforming uncertainty into a strategic advantage.

Action policies in multilingual tool-using agents are more consistent than previously thought, but only for models above a certain size threshold.

AI-AI co-creation can generate more creative and novel ideas than human pairs, reshaping our understanding of collaborative creativity in AI.

Evaluation criteria in LLM-as-judge systems are often interdependent, and RADAR reveals these hidden couplings that can skew decision-making processes.

Evaluating AI agents in a dynamic enterprise environment reveals that static snapshots miss critical context, but a new system allows for real-time, persona-driven assessments of agent performance across time.

A novel curation pipeline can optimize regression evaluation sets by maximizing capability coverage while adhering to strict query limits.

Response-only moderation maximizes usefulness, but combining input and response strategies can significantly reduce harmful exposure—revealing a critical trade-off in content moderation design.

Strong performance in static evaluations masks a critical flaw: LLMs struggle to adapt to evolving user intent during multi-turn interactions.

State-of-the-art LLMs fail to capture nuanced user preferences, lagging behind simple baselines in predicting choices in interactive narratives.

Agentic reasoning tools struggle with complex financial documents, revealing substantial gaps in their capabilities that could impact decision-making in finance.

Current LLMs only achieve 27.3% accuracy in reasoning about scientific lineage, revealing a critical gap in their compositional capabilities.

VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.

Coding agents can generate observability artifacts, but they miss key diagnostic semantics, exposing fault signals for only 13.99% of failures.

Despite high diagnostic accuracy, LLMs fail to choose valid recovery actions for over 60% of incidents, exposing a critical flaw in their operational utility.

RustMizan exposes critical weaknesses in vulnerability detection, revealing that even advanced models struggle with line localization despite decent binary classification performance.

Code LLMs can recognize incorrect instructions but still follow them, leading to irrecoverable semantic errors that defy traditional evaluation metrics.

Despite the promise of advanced AI agents, even the best performers struggle, with Codex GPT-5.5 only achieving a 42% success rate in complex healthcare tasks.

Clarifying memories can significantly boost the factual accuracy and personalization of conversational agents, while irrelevant memories lead to degraded responses.

ConflictScore reveals that language models often overlook conflicting evidence, leading to overconfident and inaccurate claims.

Just two factors can explain over 90% of a model's performance across 133 benchmarks, drastically simplifying evaluation processes.

Evaluation awareness in language models reveals a significant gap between benchmark performance and real-world safety, challenging the reliability of current evaluation metrics.

Prospective memory in LLMs is not just harder than retrospective memory; it reveals critical insights into a model's reasoning capacity and attentional robustness.

High-diversity training improves safety in VLA models, but sub-optimal trajectory synthesis still hinders task success.

Simple prompting techniques can transform LLMs into more reliable mirrors of human judgment, recovering the full spectrum of responses.

Current AI agents excel in structured tasks but falter at generating novel insights and tackling open-ended scientific challenges.

Current models struggle with hybrid interface tasks, achieving only a 41.2% success rate, underscoring a critical gap in CUA evaluation.

Selective context pruning combined with automated summarization can boost LLM performance to over 91% in complex enterprise tasks while slashing token usage and processing time.

SeClaw reveals that existing benchmarks fall short in capturing the complexities of agent behavior, enabling a more nuanced evaluation of security risks in autonomous systems.

LLMs can nail trivia in English, but stumble in Indian languages – unless you throw in some code-mixing, which magically bridges the gap.

Multilingual LLM performance disparities aren't random noise: language features and model biases systematically explain up to 92% of the variance, revealing concrete targets for improvement.

Unlock zero-shot jailbreak defense: a model's vulnerability can be predicted and patched by understanding its "behavioral geometry" relative to other models, slashing probe costs by 98%.

Exposing full reasoning traces from LLMs can actually *hurt* user performance on reasoning tasks, suggesting they're more of a distracting interface element than a helpful window into the model's thought process.

AI can help structure fuzzy concepts like "fairness" and "reasoning" into measurable specifications, potentially streamlining the evaluation of GenAI systems.

Model-generated skills can actually hurt agent performance, and bigger models don't necessarily make for better skill extractors or consumers.

Knowing which component to tweak is half the battle: directly evaluating harness optimizers via priority ranking reveals whether they're making informed decisions or just stumbling upon improvements.