Google Research

Simple ensemble methods leveraging rich textual context can outperform state-of-the-art multimodal forecasting approaches on a new benchmark, TimesX, revealing hidden vulnerabilities in existing evaluations.

CollabEval slashes evaluation uncertainty, achieving more accurate model assessments with less data by exploiting historical performance correlations.

LLMs falter on Romanized Code Mixing tasks, revealing a critical gap in their multilingual instruction-following abilities.

Directional sharpness outperforms traditional metrics in predicting model generalization, making it a game-changer for model certification.

RubricsTree transforms the evaluation landscape for personal health agents, achieving expert alignment and significant performance gains while addressing the scalability challenge in clinical deployment.

No single model dominates video embedding tasks, revealing stark contrasts in performance based on modality and task type.

LLMs reveal surprising strengths and weaknesses in analyzing security logs, with performance heavily influenced by model design choices.

Non-private synthetic data can effectively transfer knowledge from original corpora, while state-of-the-art DP methods often fail to do so, even at high privacy levels.

VLMs struggle with procedural 3D modeling, often producing flawed outputs due to API mismatches and geometric disconnections, but performance can be significantly boosted through iterative refinement.

LLM-powered honeypots can trick even frontier models into longer interactions than rule-based systems, all while costing less to run.

Even the best LLM judges miss cultural faux pas that are obvious to locals, achieving only 52% F1 score on a new benchmark.

Current remote sensing change captioning datasets miss fine-grained localized semantic reasoning, but RSRCC fills this gap with 126k change-specific questions.

Stop penalizing your ANN search algorithms for failing to retrieve irrelevant neighbors – Semantic Recall offers a more nuanced and effective way to measure retrieval quality.

Multilingual LLMs exhibit a surprising "American bias," even when prompted in other languages, and instruction tuning makes it worse.

FUSE achieves verification quality on par with semi-supervised methods, all without needing any labeled data.

Debloating tools, intended to shrink code and improve security, can actually *add* code or remove essential functionality, with dynamic methods being overly aggressive and static methods overly conservative.

Forget KL divergence – this work shows you *can* reliably evaluate generative models with finite samples, but only if you use the right metric (IPMs with bounded test classes).

Safety fine-tuning might inadvertently be stripping LLMs of their ability to understand non-human minds and entertain spiritual beliefs, even while preserving Theory of Mind.

MLLMs are surprisingly prone to hallucinating subtle details, especially when asked about the absence of specific attributes or relationships within an image.

Reasoning unlocks factual knowledge in LLMs, but beware: hallucinated reasoning steps can poison the well.

LLM-powered diagnostic AI is ready for prime time: a real-world clinical trial shows it's safe, patients love it, and doctors find it useful.

Finally, a framework to quantify AI's cultural intelligence, moving beyond ad-hoc cultural benchmarks to a systematic, extensible, and theoretically grounded approach.

Gemini 3 Deep Think can now autonomously solve a majority of problems in a challenging math competition, signaling a leap in AI's mathematical reasoning capabilities.

LLMs still struggle with infrequently occurring knowledge, and this paper provides a structured framework to understand why, how we can fix it, and what the implications are for responsible AI.