Tsinghua AI

Current LLMs falter in complex deliberative collaboration tasks, revealing critical gaps in their reasoning capabilities even when aided by external tools.

Existing text-to-image models struggle to capture individual aesthetic preferences, but PIPBench reveals critical gaps in their performance that could redefine personalized image generation.

Many robotic policies that seem successful in manipulation tasks actually compromise safety, with SoftVTBench revealing a stark contrast between goal completion and physical safety metrics.

AgenticDataBench reveals that LLM-based data agents can be rigorously evaluated across diverse real-world scenarios, highlighting their strengths and weaknesses in handling complex data tasks.

GPT-5.5 not only tops the leaderboard in policy evolution but also reveals critical insights into how agents can optimize performance through strategic feedback utilization.

LLMs show significant vulnerability to logical fallacies, with distinct profiles of resilience that could inform future model training strategies.

Current interactive world models fall short, with none passing the rigorous tests of WorldRoamBench designed to assess long-horizon stability across action, vision, physics, and memory.

Even the most advanced LLMs struggle with consistent rubric verification, revealing substantial noise in scoring outputs across complex agentic scenarios.

Evaluating LLM agents in microservice failure diagnosis reveals that traditional outcome-based benchmarks miss critical reasoning processes, which these new datasets effectively capture.

HarmVideoBench reveals that existing benchmarks miss critical layers of harmful video understanding, while a new method boosts model accuracy by over 20%.

Expressiveness preservation in speech-to-speech translation remains a significant hurdle, with systems scoring poorly on emotional and nonverbal fidelity despite achieving high translation accuracy.

Despite high benchmark scores, SOTA semantic code clone detectors falter in real-world scenarios, revealing a reliance on shortcut learning over genuine semantic equivalence.

Strong proprietary models falter in grounding their predictions, revealing a critical flaw in current VideoQA systems that could reshape evaluation standards.

Isolated assessments may mask biases, but comparative evaluations can unleash hidden discrimination in LLMs, especially as model sizes grow.

No single memory architecture is best for all tasks; performance hinges on how well memory structures align with specific workload challenges.

Current vision-language models falter in streaming interaction understanding, with alarming mis-calibration leading to confidently incorrect predictions.

MLLMs falter in fine-grained interpersonal reasoning, but integrating visual cues and social roles can dramatically boost their performance.

High-diversity training improves safety in VLA models, but sub-optimal trajectory synthesis still hinders task success.

CUAs can achieve a 73.7% success rate on complex macOS tasks, but the secret to their performance lies in skill libraries, not just framework design.

LLMs show no detectable self-preference, rejecting valid corrections to their own drafts at the same rate as neutral judges.

Advanced RS MLLMs struggle with negation, but a novel learning method can dramatically enhance their understanding using minimal unlabeled data.

I2V models not only excel at dynamic editing but also provide a unique lens for diagnosing errors in Human-Object Interaction tasks.

HandTouch outperforms existing tactile encoders, achieving an 85.23% Recall@5 in similarity retrieval—an improvement of over 10%—demonstrating the power of combining egocentric vision with tactile data.

Structural fidelity in webpage generation drops significantly with length, revealing that visual appeal doesn't guarantee functional effectiveness in VLMs.

Pairwise evaluations of LLMs can yield cyclic inconsistencies, but a novel prompt perturbation framework can stabilize rankings and enhance interpretability.

Models with similar success rates can exhibit vastly different strengths and weaknesses, revealing the hidden complexities of mobile manipulation capabilities.

Semantic acceptance rates can be misleading, with up to 44.2% of models failing to prevent observable harm even when they pass initial checks.

Advanced planners still exhibit critical safety failures, with FluidTest uncovering new threats in over 65% of evaluated trajectories.

Despite high retrieval rates, LLMs miss over 47% of relevant studies in meta-analysis, exposing a critical gap in their systematic reasoning capabilities.

LLMs miss over 50% of errors in human-written code, but with test-time scaling, they can identify issues in more than 90% of cases—if you can afford the compute.

Current vision-language models struggle with process understanding in robotic manipulation, but targeted post-training can yield significant improvements.

Adapter design can make or break coding performance in OpenClaw-style agents, with a full adapter boosting success rates by over 50 percentage points.

Automation in benchmark construction shifts costs from creation to governance, revealing a hidden complexity in evaluating embodied intelligence.

Current autonomous AI agents are alarmingly unprepared for real-world adversarial attacks, often missing critical vulnerabilities in dynamic environments.

Even the best LLMs struggle with Olympiad-level combinatorics, achieving only 65.4% on a benchmark designed to expose their reasoning limitations.

VLM agents exhibit vastly different skill evolution patterns, revealing that initial performance scores can be misleading without considering improvement dynamics.

Despite the advancements in multimodal agents, even the best models struggle with interactive spatial reasoning, achieving only a 17.4% success rate in complex real-world tasks.

LexRubric reveals that even state-of-the-art LLMs struggle with open-ended legal tasks, exposing critical gaps in their contextual understanding and reasoning abilities.

Current models struggle with hybrid interface tasks, achieving only a 41.2% success rate, underscoring a critical gap in CUA evaluation.

Geometry dictates model performance in relational learning, with rankings shifting dramatically across curvature regimes, challenging the reliability of standard evaluation practices.

The hardest AI tasks remain largely unsolved, with current models achieving only a 2.6% success rate on economically valuable workflows.

Streaming spatial intelligence remains a significant hurdle for multimodal LLMs, with top models trailing human experts by 27 points in allocentric mapping tasks.

ClinEnv reveals that LLMs struggle significantly with management decisions in clinical scenarios, achieving only 0.17 F1 for these critical actions despite better performance in diagnosis.

No current MLLM can reliably issue timely safety warnings, with performance sharply varying across domains and a troubling trade-off between recall and false positives.

Current LLM judges show a troubling reliability gap in long-form evaluations, raising questions about their effectiveness in real-world applications.

Evolving coding problems can restore meaningful evaluation metrics for frontier models, revealing their true capabilities and enabling self-improvement.

Open-source E2E models outperform Cascade systems in understanding heavy dialects, but struggle significantly with low-resource languages, revealing critical gaps in current speech model architectures.

Decompilers might produce readable code, but good luck getting it to actually *work* – a new benchmark reveals a massive gap between recompilability and functional correctness.

Ditch the textual explanations: symbolic outputs like bounding boxes are the secret sauce for boosting multimodal verifier performance.

LLM trading agents might seem profitable, but a new benchmark reveals their returns are mostly just riding market trends, not actual stock-picking skill.