Eval Frameworks & Benchmarks - Weekly Roundup

GPT-5.4 outperforms smaller models in synthesizing transformation rules, achieving high applicability rates while revealing the limitations of LLMs in complex software evolution tasks.

DataFlex-RL, an evaluation platform for comparing choices under a common GRPO recipe, is introduced, finding that changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

Frontier multimodal models that master complex single-image visual reasoning collapse to sub-35% accuracy when asked to detect basic low-level noise and texture differences between two images.

EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules.

A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

DataFlex-RL, an evaluation platform for comparing choices under a common GRPO recipe, is introduced, finding that changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

Frontier multimodal models that master complex single-image visual reasoning collapse to sub-35% accuracy when asked to detect basic low-level noise and texture differences between two images.

EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules.

A single, architecture-independent mechanism behind the conditions where diff-based generation does win is identified: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in the authors' dataset.

Diverse Skill Routing is proposed, a diversity-aware reranking framework that uses a Determinantal Point Process to balance relevance and non-redundancy and improves recall and full coverage over a strong pointwise reranking baseline, with larger gains on multi-skill queries.

Srijika is presented, a system for producing installable OpenType fonts for nine Brahmic scripts: Devanagari, Tamil, Bengali, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, and Odia, and a negative-results catalogue covering failed conditioning, objective choices, and data-hull limits of reference-guided restyling.

Even frontier coding agents collapse to a 23.9% success rate when tasked with end-to-end agent construction under realistic client constraints, falling nearly 60 percentage points short of human engineers.

VLMs that reliably refuse entirely invalid prompts systematically fall apart when unanswerable, infeasible, or unsafe sub-claims are embedded inside otherwise benign compound queries.

Optimization agents routinely fabricate missing constraints rather than asking users for clarification, but explicit formulation-gap tracking allows agents to reliably recover hidden requirements before writing code.

Production LLM trading agents trail basic retail benchmarks by nine percentage points in win rate, largely because mechanical scaffolding and volatility-blind sizing—not strategic reasoning or prompt design—almost entirely dictate live agent behavior.

WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.

Scaling task complexity across 280 standardized manipulation variants reveals that today's leading VLA models fundamentally collapse under fine-grained spatial ambiguity and memory-intensive sequential planning.

Occamy-1.0, a cost-efficient co-work model obtained by further training the post-trained Qwen3.6-35B-A3B checkpoint, is presented and placed at the low-cost knee of the observed cost--performance Pareto frontier.

Numerical reasoning in HTN planning just got a major upgrade with a new SMT-based encoding that sets a competitive baseline for future advancements.

The tool-call rate can swing from zero to hundreds based solely on the interface used, revealing a hidden layer of performance that challenges conventional evaluations of model efficacy.

Seemingly benign user requests frequently clash with unstated personal constraints, yet current LLM assistants consistently fail to retrieve the implicit knowledge-base evidence required to trigger appropriate refusals.

LLMs can match or outperform traditional CKD screening methods with minimal examples, but their stability diminishes with increased input complexity.

Performance rankings in continual knowledge updating can flip dramatically based on evaluation timing and replay capacity, challenging conventional wisdom.

Only one out of 178 problem-model cells shows prefix-limited reasoning, while most LLMs outperform restarts, highlighting that compute compression is the key driver of performance.

Increased generative-process diversity in language models significantly reduces correlated failures, a factor overlooked by traditional semantic similarity assessments.

ESPO not only boosts accuracy by nearly 4 percentage points but also slashes prompt length by nearly half, redefining efficiency in prompt optimization.

Functional success in coding agents is misleading—over 34% of patches that pass tests still fail to meet critical review constraints.

EquiReview-R reduces major overcritique by nearly half while ensuring critical issues are not overlooked, challenging the notion that quantity of criticism equates to quality.

A standardized protocol for LLM brand recommendation auditing reveals that just 15 iterations can significantly enhance reliability, challenging existing practices in the field.

BLEU-4 may mislead SLT progress, masking the true performance differences between gloss-free and gloss-supervised systems.

Existing hallucination detection methods falter in the nuanced landscape of scientific peer reviews, revealing a critical gap in ensuring review reliability.

Single-pass annotation can miss critical factual errors in chatbot responses, but a multi-perspective approach reveals a more comprehensive picture of accuracy.

No single large model dominates in parametric CAD modeling, revealing significant trade-offs in executability and visual-semantic identity.

The resulting 82M-parameter model, Wayu-Paxa-TTS-Edge, enables on-device Thai TTS without reference audio and achieves the lowest pause-placement error and intra-word pause rates among the three systems.

Conversational agents may be missing critical context, with implicit queries revealing substantial retrieval gaps that traditional benchmarks overlook.

Multilingual LLMs suffer substantial reasoning failures when faced with structurally altered inputs, revealing a critical vulnerability in their design.

No single large language model dominates idiom comprehension in Bangla, revealing nuanced strengths and weaknesses across different tasks.

Medical image reasoning claims are often inflated by shortcuts, with accuracy plummeting to 5.21% when these cues are mitigated.

Flawed benchmarking practices in microarchitectural side-channel research are more common than expected, with each paper averaging 5.5 significant issues that undermine reproducibility and relevance.

OCR-EDR transforms OCR error analysis into actionable repairs, achieving a remarkable 94.78% diagnostic accuracy and a 30.99-point boost in formula performance.

R2S-Eval reveals that automated evaluation can outperform manual success counting by capturing nuanced differences in robot behavior quality.

Grounded in real-world evidence, GPS-Bench reveals that fine-tuning LLMs on historical policy data dramatically enhances the accuracy of multi-agent simulations in predicting policy outcomes.

Machine translation benchmarks have functionally saturated, but pairing human-authored failure cases with deterministic verification rules reveals critical multimodal blind spots that automated metrics consistently miss.

Language-model judges may be fundamentally unreliable, with repeatability scores failing to meet established benchmarks, jeopardizing the integrity of AI evaluations.

Safety compliance can vary by over 28 points among LLMs with similar predictive accuracy, revealing hidden risks in flight predictions.

Seizure detectors may perform well on clean data but can falter dramatically under real-world conditions, revealing a critical gap in pre-deployment assessments.

Traditional single-session assessments can mislead clinicians by failing to accurately track worsening depression trends, underscoring the need for longitudinal approaches.

Current PII detection systems fail dramatically under real-world conditions, with each architecture revealing unique vulnerabilities that standard benchmarks overlook.

One in four AI-generated patches may simply replicate historical fixes, raising serious questions about the reliability of current vulnerability patching evaluations.

MLLMs struggle with real-world industrial measurements, achieving only 25.7% accuracy in a new benchmark that simulates authentic operational challenges.

Current evaluation metrics for SVG generation miss critical errors, but SVG-Score reveals how to align machine outputs with human judgment effectively.

Safety evaluations of LLMs vary dramatically across languages, revealing that some harmful content is far more susceptible to manipulation than previously understood.

A novel benchmark dataset reveals that a multi-stage RAG pipeline can dramatically boost financial question answering accuracy by over 34 percentage points.

VLMs struggle with robot task evaluations, achieving only 0.77 mean balanced accuracy, and even fine-tuned models often underperform compared to general-purpose counterparts.

Strong benchmark scores can mask critical failures in LLMs' ability to apply individual content moderation criteria effectively.

CreaEval's dual-phase approach not only mitigates biases in LLM evaluations but also boosts performance by over 22% in complex creativity tasks.

Multimodal LLMs fail to grasp 39.3% of essential legal queries in Bengali land records, revealing a critical capability gap in AI's understanding of complex, domain-specific tasks.

The strongest LLM only achieves a macro-F1 score of 67.3 on a benchmark designed to capture the complexity of opposing emotions, revealing significant gaps in current emotion recognition systems.

Large language models can surpass human performance in understanding frame semantics, revealing their advanced implicit comprehension abilities.

VLMs exhibit striking variability in their ability to correct repeated false premises, with some models showing persistent inaccuracies that could mislead users.

Robustness claims in language models can be misleading when based solely on output behavior, as perturbations reveal complex, multi-level effects that traditional metrics overlook.

LLMs generate plausible but ungrounded code 60% of the time, revealing a critical gap in their reliability for programming tasks.

Off-the-shelf LLMs miss nearly 53% of critical requirement issues, raising serious questions about their reliability in engineering contexts.

Grading multi-topic essays without explicit labels can achieve unprecedented accuracy through graph-augmented retrieval techniques.

GPT-5.4 outperforms smaller models in synthesizing transformation rules, achieving high applicability rates while revealing the limitations of LLMs in complex software evolution tasks.

Today's top video generators score near 0.8 on standard benchmarks like VBench, but none surpass 0.42 when tested on whether two co-occurring objects obey the same basic laws of physics.

Practical non-invasive BCIs cannot require days of user calibration, and this benchmark forces models to tackle word-level MEG decoding under strict 10-to-40-minute subject adaptation budgets.

Frontier LLMs will ruthlessly crush animals to minimize fuel costs unless explicitly told not to, revealing that safety alignment completely collapses into 84%+ kill rates under concrete economic trade-offs.

High Pass@1 masks a critical failure mode in frontier code models: models routinely ship bloated, unreviewable diffs for trivial bugs, an issue that scaling cannot solve but RL post-training can directly optimize.

Propagating local edits across multi-turn artifacts does not require expensive sequential reflection loops—reranking just three parallel samples reliably yields up to a 9.7% consistency boost across open and frontier models.

Expanding an agent's harness with new tools and skills often degrades its performance on tasks it previously solved, exposing a critical failure mode of "harness-induced forgetting."

Mutation analysis as an adequacy metric for kernel-benchmark oracles is introduced: deterministic rules inject 10 compilable faults into verified CUDA implementations of 188 KernelBench problems, 7{,}384 of them with an independent kill witness; any test protocol is scored by the fraction it detects.

LLMs struggle to generate robust travel itineraries, with substantial gaps in handling delays and crowd dynamics revealed by UTP-Bench.

Misleading assertions corrupt LLM medical reasoning more than fabricated evidence, yet are disclosed less often, exposing a critical flaw in model transparency.

TFMs may excel at table-filling tasks, but they fundamentally misunderstand the underlying physics, lacking the ability to represent essential concepts like physical units.

AI-derived measures can recover critical individual and group-level effects, challenging the traditional reliance on survey data for social and occupational insights.

After two decades, a definitive independent set for the frb100-40 benchmark proves its maximum independent-set size is 100, but no search strategy offers a heuristic advantage.

Automated techniques can enhance alignment evaluation realism, yielding greater insights into model behavior than traditional methods.

A pseudo-video spatial-memory Transformer achieves perfect validation accuracy on complex tasks, revealing that structured working memory can outperform mere latent capacity in predictive modeling.

Estimating optimal BER and AUC from soft labels reveals significant insights into model performance that traditional accuracy metrics can obscure.

Mislabeling in encrypted traffic benchmarks can lead to a staggering 40% drop in classifier accuracy, challenging the validity of existing models.

Stacked, a novel supervised method, significantly enhances hallucination detection in LLMs, outperforming alternatives in nearly half of the tested scenarios.

MLLMs struggle with multi-hop reasoning in document-style images, achieving only 62.83% accuracy compared to over 90% for human annotators.

Even top-performing text-to-image models struggle with generating coherent visual metaphors, revealing a critical gap in their creative capabilities.

Agents in CivBench under-monitor strategic states and frequently fail to act on their own planning commitments, revealing critical gaps in tool-mediated decision-making.

Multi-turn jailbreak attacks reveal that framing requests as actionable can significantly reduce model vulnerability, highlighting a critical leverage point in AI safety.

SCX Router achieves a 1.5% performance boost over the best fixed model by intelligently selecting the most suitable LLM for each task in real-time.

Achieving a staggering 99.50% acceptance rate in synthetic dialogue generation reveals the transformative power of feedback-guided refinement in meeting complex communicative constraints.

Two AI systems with nearly identical performance can have drastically different human oversight needs, revealing hidden costs in deployment readiness.

Achieving up to 80% judgment reuse, this method slashes evaluation costs and streamlines retrieval model selection in real-world applications.

Despite advancements in OCR and vision-language models, critical page-level leakage of PII remains dangerously high, reaching 0.968 even with improved detection methods.

Output format can distort perceived model capabilities, with a 40-point accuracy gain in one format vanishing in another.

LLMs can significantly enhance user interactions by mastering question clarification, as revealed by a novel tri-agent evaluation framework that benchmarks their performance.

Withholding just one terminal receipt can render coverage inconclusive, challenging the reliability of agent evaluations in practice.

Machine Learning-based Intrusion Detection Systems may appear effective, but they often fail to detect unseen attacks, revealing critical vulnerabilities in their design and evaluation.

MLLMs may rank faces similarly to humans, but they systematically overrate attractiveness, raising questions about their reliability in aesthetic assessments.

VIPS reveals that integrating vehicle and infrastructure observations can significantly enhance the robustness of autonomous driving evaluations without the pitfalls of traditional simulation methods.

Causal-drive trajectories reveal a surprising shift in VLM reliance from visual cues to generated prefixes, enhancing our understanding of multimodal generation dynamics.

EarlyEval can cut evaluation costs by up to 44% without sacrificing accuracy, revolutionizing how we assess LLM agents.

User feedback can significantly enhance LLM performance, but current evaluation methods often overlook its benefits, leading to misguided assessments of model improvements.

A 7B model can boost source-grounded speech planning accuracy from 68.4% to 91.9% by addressing citation legality before synthesis.

Language models exhibit substantial incoherence in probabilistic forecasts, with irrelevant details amplifying errors by an order of magnitude.

SHELF reveals that sparse methods can outperform traditional approaches in bibliographic tasks, challenging assumptions about model performance consistency.