Eval Frameworks & Benchmarks - Weekly Roundup

A novel analytical model termed Exponential Pixelating Integral is introduced for the automatic detection of infections in Chest X-Rays, registering a higher classification accuracy and F1 scores than its peers by registering a higher classification accuracy and F1 scores.

Inspired by Mixture-of-Experts (MoE) principles, Evolvable-Substrate HyperNEAT is partitioned into non-overlapping spatial segments, each assigned to a separately evolved specialist network, which forces evolution to discover features across the entire image.

Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction, which leads the compared models on MMAU, Llama Questions, and Speech CMMLU.

StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop, achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time, and resolves the tension between deep deliberation and latency via Think-While-Speaking.

This work presents RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings.

Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction, which leads the compared models on MMAU, Llama Questions, and Speech CMMLU.

StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop, achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time, and resolves the tension between deep deliberation and latency via Think-While-Speaking.

This work presents RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings.

This work introduces StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation, and achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems.

Continual Search is introduced, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence within multiple benchmark suites and model families, demonstrating that effective search supersedes raw model scale.

UFO is proposed, the first unified framework for omni-condition alignment simultaneous evaluation, and UFO-Bench is presented, a dedicated benchmark designed to holistically evaluate the performance of existing customization models under the diverse mutual interactions of textual and visual conditions.

This work presents ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency, and establishes an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation.

This work introduces Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts, and establishes minimal intervention as an effective paradigm for enhancing on-policy RL.

NGU, a simple adaptive sampling method that keeps generating samples for a problem until one is correct, is introduced, a simple adaptive sampling method that naturally uses fewer samples to filter out easy problems and allocates more compute to solving harder problems.

It is shown that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training.

eConv-TasNet is proposed, an efficient variant of Conv-TasNet that improves both effectiveness and efficiency without relying on resource-intensive modules, and achieves competitive performance compared with state-of-the-art methods while requiring significantly fewer parameters and lower inference cost.

UniH3, a novel framework that Unifies Hierarchical Homogeneity and Heterogeneity and Heterogeneity for all-in-one medical image restoration, is proposed and extensive experiments on two large-scale benchmarks demonstrate that UniH3 achieves state-of-the-art performance on both all-in-one and single-task medical image restoration.

The Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen, is proposed, which improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation.

This work proposes RDDMPI, a conditional residual diffusion framework that operates directly in residual space, and reformulates probabilistic imputation as a baseline-residual decomposition, where a pretrained model captures the dominant signal and a diffusion process models the residual uncertainty.

Results show that structured schema context and lightweight prompting can substantially reduce reliance on fine-tuning for scalable conversational access to KGs, though closing the remaining gap to fully fine-tuned approaches will likely require reducing dependence on curated question-query exemplars.

Non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization, and transfer from non-language data is less efficient than additional language data.

This work presents FreeFlow, a hierarchical transformer built without any flow-specific components, using instead a single feed-forward encoder--decoder, achieving state-of-the-art results on major benchmarks, including Sintel, KITTI-2015, and Spring.

T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier by executing each task's own verifier.

TART is the first framework to generate guitar tablature with both fingering and expressive technique annotations directly from guitar audio, and is the first framework to generate guitar tablature with both fingering and expressive technique annotations directly from guitar audio.

It is found that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth, not only on device bandwidth.

A quantum approach to spectral feature extraction from the density of states (DOS) of a problem-dependent Hamiltonian is developed, and applied to machine learning on signed graphs, and opens routes towards quantum applications in social network balance analysis, spin-glass studies, correlation clustering, and protein-interaction networks.

Inspired by Mixture-of-Experts (MoE) principles, Evolvable-Substrate HyperNEAT is partitioned into non-overlapping spatial segments, each assigned to a separately evolved specialist network, which forces evolution to discover features across the entire image.

E-CONAN benchmarks that are composed of sentences pairs from various sources, derived from a combination of sources, offers a broader and more robust assessment compared to XNLI and ArNLI, and are used to evaluate state-of-the-art multilingual pretrained models using zero-shot classification.

This work revisits RAG compression from a data-mining perspective by aggregating historical query--document--model interactions into reusable evidence views and proposes Reusable Evidence View Aggregation (REVA), a framework that mines the target generator's historical attention traces into a document-keyed, budget-agnostic score store.

This paper extends CONES to allow for loss functions $f_t'$s to also change over time, and shows that any online algorithm with sublinear {\it anytime} regret has a movement cost of $\Omega\left(\log T\right)$.

Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.

AssayLoop is introduced, a sequential experimental design framework combining AssayFormer, a transformer-based amortized acquisition policy trained across historical screens to adapt from experimental feedback, with LLM-derived biological priors through an adaptive handoff to demonstrate the value of learning acquisition policies across historical experiments and combining them with broad biological priors for efficient adaptive hit discovery.

The central finding is that aggregate WER hides code switching behavior, and the best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric.

COBRA-Skills is introduced, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space and remains robust to changes in the agent harness and performs effectively when the target model itself is used for skill generation and refinement.

WorldVIEW, a multilingual benchmark of 8,960 prompts across 15 languages and 31 language-context pairings, is introduced, and the revision layer itself is identified as a previously undocumented, causal source of this stereotyping.

The canonical region graph is proposed, an input representation whose nodes, features and coordinate frame are derived from the solid itself and show theoretical invariance guarantees on repartitioning and rigid motions and matches the strongest baseline on standard benchmarks, and is stable under every perturbation the authors test.

This paper introduces Magenta, a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof, which achieves 100% accuracy across all evaluated olympiad benchmarks.

Results are reported, showing an inverse relationship between benign and unsafe capability, strong evidence that baseline safety guardrails do not lead to downstream safety guarantees in the RAG case, and model-specific support for previous findings that even benign documents can lead to unsafe generation in retrieval-enabled systems.

It is shown that the appearance of nodules below six millimeters is highly regular, and proposed GRIPNet (Gaussian Radial Intensity Prior Network), a detector in which every module maps to a measurable property of the intensity distribution, is proposed.

This work proposes two approaches of code-mixed generation using parallel sentences of three languages and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages for language identification in code-mixed settings.

This work proposes Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM, and introduces VWG-Bench, a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.

This work constructed a large-scale Arabic Speech Question-Answering (SQA) corpus comprising over 1.5 million training samples to allow instruction tuning over a broad range of core speech tasks and designed an evaluation framework featuring diverse tasks and tailored metrics.

Whether medical research is keeping pace with the systems it evaluates is asked, and Rigour and currency are in tension, and that tension reflects model selection rather than research timelines.

This work benchmarked ten classifiers spanning linear, tree-ensemble, neural, glass-box, and tabular foundation classes for prevalent myocardial infarction in 442,067 respondents of the 2022 Behavioral Risk Factor Surveillance System across five feature tiers of decreasing leakage risk.

This work analyzes over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identifies 12 recurring noise patterns, which are group into formatting and content noise, and constructs TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs.

A unified evaluation framework testing gradient and loss-based balancing strategies under controlled settings, and a theoretical diagnosis explaining why these methods fail, as they conflate fitting speed with discriminative contribution are provided.

ProMediConv is introduced, a novel benchmarking framework that models mediation as a proactive, multi-stage, and party-aware dialogue process incorporating 11 mediation strategies and four party behavior pattern (BP) states and proposes MAD (Mean Attribute Difference), a fine-grained metric that captures BP shifts throughout the dialogue.

ReGround is introduced, a large-scale dataset for reviewer comment grounding that links 10,267 reviewer comments to 16,274 evidence in the original anonymous submission of 3,656 papers and exposes grounding reviewer comments as a difficult and practically important problem for scientific document understanding.

This work introduces AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains, and uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics.

A controlled benchmark of membership inference vulnerability for text classification on the GLUE SST-2 sentiment dataset is presented and lightweight training adjustments can improve the privacy-utility trade-off without complex defenses.

Financial NLP has a standard workflow: validate a sentiment tool against human labels, then trust it to extract market signal, but it is found that the relationship between construct and predictive validity depends on the sampling convention and score representation.

A three-tier evaluation framework for unsupervised narrative label generation is introduced: recovery (against a corpus's own taxonomy), mining (against external label sets), and discovery (without predefined labels).

Atlas is the first to demonstrate verifiable graph-based search at scale, proving a query in under a second on the SIFT1M benchmark and in 2.0 seconds at 100 million vectors, while maintaining the recall of plaintext HNSW and revealing nothing about the index beyond the result.

RCL (Retrieval-Confidence Layer), a lightweight module inserted between retrieval and generation that combines a call-graph-derived structural coverage score with a novelty score estimating a query's dependence on knowledge outside the model's prior, to detect insufficient retrieval before generation occurs.

ChurnBench is presented, an open-source benchmark that generates a four-source enterprise data fabric as a timeline rather than a snapshot, and it is found that when a system refreshes on a schedule, cache age does not predict staleness.

LLMVul, a vulnerability-labeled dataset of LLM-generated C/C++ functions mined from real production repositories, enables reproducible research on vulnerability detection, security evaluation of LLM-generated code, and analysis of vulnerability patterns in AI-assisted software development.

A protocol that makes the referee's effect measurable and whose answers cannot be narrated afterwards is reported, which finds that the arm instructed to specify and verify cost more on all five components, and by a practical margin.

SemVerBench is introduced, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo), and six frontier models are evaluated: Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar).

This work proposes Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework that turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length.

This work proposes Tri-DehazeGS, a scene--medium decoupled Gaussian Splatting framework, which represents the clean scene with Gaussian primitives, models the participating medium using an independent view-shared tri-plane field, and composes hazy observations through a physical scattering model.

Empirical results show that CamPilot outperforms state-of-the-art text-to-movie generation methods on cinematographic control and quality, highlighting the impact of professional camera design on movie generation.

This paper proposes OmniKVQuant, a training-free framework that enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks and provides a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built.

This paper proposes MOCO, a novel diffusion-based framework capable of processing multiple simultaneous inputs, including speech audio, text descriptions, and trajectory data, to generate coherent and lifelike motions without requiring aligned multimodal data.

A mechanistic analysis of the LW-DSC metric is provided that explains why recall-recovering post-processing carries low risk whereas component deletion does not, and thirteen negative results spanning loss engineering, alternative backbones, and inference-time settings are reported.

It is shown that modality-unified prototypical contrast facilitates better modality invariance by jointly and simultaneously optimizing similarity relation within and across-modality.

HighPerViT is presented, a compact vision-only architecture that injects an explicit second-order statistical prior into a transformer-based recognition pipeline, and integrates this token with first-order spatial representations through Perceiver-style latent distillation, supporting explicit statistical tokenization as an effective and robust design principle for texture-centric visual recognition.

IMLE-VLA is introduced, which replaces the iterative action head with a single-step conditional generator trained via conditional Implicit Maximum Likelihood Estimation (cIMLE), and promotes multimodal action coverage, avoiding the mode collapse of naive regression heads while eliminating multi-step sampling entirely.

EgoGenEval introduces EgoGenEval, a geometry-grounded, pose-free benchmark designed to evaluate the physical consistency of visual generators under ego-motion, and shows that pairwise supervision does not reliably improve camera-motion grounding and scene-state preservation together.

A novel analytical model termed Exponential Pixelating Integral is introduced for the automatic detection of infections in Chest X-Rays, registering a higher classification accuracy and F1 scores than its peers by registering a higher classification accuracy and F1 scores.

Frozen predictive world-model states as reusable features for causal, transferable, and low-overhead execution monitoring for reliable robot deployment are supported.

AudioICL-Bench is introduced, a diagnostic benchmark whose per-episode rules are resampled so that no correct answer is recoverable from prior knowledge, and its nine tasks are organized along two axes that separate what must be learned from demonstrations from what must be perceived in the signal, enabling failures to be attributed to either source.

Sewer-Transformer-ML is developed, a hierarchical vision Transformer with multi-level feature fusion together with two lightweight architectures, Sewer-MobileNet-ML and Sewer-Mobile-TransNet, for resource-constrained inspection scenarios and provides a computational basis for automated sewer inspection, lightweight model design, and adaptation across civil infrastructure inspection platforms.

This report presents a multi-view and confusion-guided ensemble framework for the Synthetic Image Attribution Challenge of the DLMMDD Workshop at ICANN 2026, and introduces a dedicated binary expert classifier that is selectively activated under low-confidence conditions.

Omni-Modal Gait Recognition is formulated, which unifies single-modal, cross-modal, and multi-modal recognition within a shared identity space and remains competitive with task-specific experts in many shared settings and extends to higher-cardinality fusion unavailable to fixed-pair models.

The result is a practical way to separate two failure modes that are usually mixed together: a weak selector versus an information channel that cannot support the desired decision in the first place.

VP-CONTROL, a runtime-assurance design and deterministic benchmark for cost-aware commit gates, is presented, motivating explicit evidence lineage, cost-aware selection, and commit-time enforcement, while exposing the limits of approximate calibration and local-tool generalization.

CausalArena is introduced, a unified and evolvable benchmark for causal discovery under a common protocol, and substantial ranking shifts across SCM families and protocols are revealed, showing that strong performance in one benchmark regime does not reliably transfer to others.

This work reifies the input graph by reifying the input graph: every fact becomes a node, connected to its subject, object, and relation type through a fixed vocabulary of six meta-relations, with relation types as anonymous shared nodes rather than model parameters.

GenQAS is introduced, a tensor network-guided RL framework that combines a fixed matrix product state warm-start with prioritized generative replay and can mitigate sample starvation in quantum architecture search and support more resource efficient circuit discovery.

A multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection is presented.

This work introduces AmazonSWE, a dataset for training and evaluating large-scale spatiotemporal graph imputation methods that integrates processed satellite altimetry measurements from a range of sources, including the recent wide-swath SWOT sensor.

AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes is introduced, which is scalable, model-agnostic, and straightforward to integrate into existing training pipelines.

Looped flows are proposed, an approach that allows solving harder problems by spending more computation through a finer temporal grid and enables multiple valid predictions from different initial noise samples.

The proposed Koan-Safe, which combines a prompt-only intent parser, a replaceable generator, and structural repair with default safety parameters, is proposed, which supports explicit trade protections and execution-based evaluation, while distinguishing declared safety from a general guarantee.

HerALD (High-fidelity Exemplar Retrieval with Adaptive Landmark Distillation with Adaptive Landmark Distillation), a gradient-free graph condensation framework that adapts the node scoring and feature selection in the condensation pipeline to the graph's measured heterophily, is proposed.

This paper model MTO as a sequence of sequential transfer optimization problems, concentrating evaluations on a single target per iteration, and proposes a likelihood-informed task prioritization mechanism to maximize transfer utility by identifying the task most likely ready for knowledge integration.

This work proposes Test-Time Adaptation via Stability-Aware Confidence Optimization (TASCO), a framework that incorporates local stability into confidence-based test-time adaptation while keeping the LLM frozen and improves reasoning accuracy and token efficiency across diverse LLMs and reasoning benchmarks.

It is found that all linear performance leaders effectively add only a nuclear-norm based regularizer, or a Frobenius-norm based regularizer, or a Frobenius-norm based regularizer.

Calibration-Aware Uncertainty Cascades is proposed, a simple post-hoc framework that independently calibrates each model's confidence and selects deployment policies using validation data and shows theoretically that calibration gives confidence thresholds an explicit selective-risk interpretation, whereas uncalibrated scores offer no comparable reliability guarantee.

NovGauge is presented, a human-anchored benchmark for fine-grained novelty assessment diagnosis, and a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support is proposed.

This paper proposes SWRouter, a Similarity-Contractive Window Router for multi-turn large language model routing that combines a similarity-based context segmentation mechanism for prompt construction with a dual-metric evaluation framework that decouples construction accuracy from router performance.

Sci-MMR is introduced, a benchmark for multi-step evidence-grounded scientific reasoning built on structured argument graphs linking scientific claims, citation-grounded knowledge, visual evidence, and supporting regions, and it is found that current answer-centric benchmarks substantially overestimate the evidence-grounded reasoning capabilities of multimodal research agents.

Whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform is studied to reveal limitations in physical reasoning that conventional answer accuracy can overlook.

This work introduces MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots, which performs better on reasoning than on planning, and the best-performing model remains far below observed human performance.

A Database Normalization Benchmark (DNBENCH), comprising 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF, and proposes Multi-Agent Reasoning for Schemas (MARS), which separates evidence extraction, violation diagnosis, and decomposition planning from schema generation and verification.

BenchShield is presented, a model-backed instrumentation layer for reward integrity in LLM-agent evaluation that grounds detection in a finite lifecycle model of an evaluation's reward-relevant events and achieves 96% accuracy in detecting reward hacking from infrastructure-side evidence.

This work introduces MAPLE (Memory-Augmented Planning with Language and Evolution), an agent for maintaining optimization problems through successive natural-language requests and introduces NLDO, a benchmark of 15 trajectories and 180 updates spanning selection, scheduling, rostering, routing, and cloud-resource placement.

It is concluded that anonymization is not a one-size-fits-all solution and must be co-designed with the model and task in mind to balance privacy and utility effectively and provide a crucial baseline for developing more robust, privacy-aware AI systems.