Search papers, labs, and topics across Lattice.
This paper introduces JudgeFit, an iterative refinement procedure that creates tailored evaluation taxonomies for vision-language models (VLMs) by leveraging their unique internal representations of physical phenomena. By prompting each VLM to identify physics errors in videos and clustering these descriptions, the method generates an initial taxonomy, which is then refined through calibration against human ratings and iterative diagnostics. The results show that this per-VLM approach yields a mean relative improvement of approximately 32% over a global evaluation schema, revealing model-specific blind spots that traditional rankings overlook.
Tailoring evaluation taxonomies for each vision-language model reveals a 32% improvement in performance and uncovers unique model blind spots that global assessments miss.
Maintaining physical consistency in video generators and world models increasingly relies on vision-language models (VLMs) as automated judges that provide reward signals, ranking decisions, and data-filtering criteria. Yet VLMs differ substantially in training data and architecture, encoding physical phenomena through distinct internal representations. A single global evaluation schema therefore gives every VLM the same axes of competence, regardless of what each can actually perceive. We propose JudgeFit, an iterative refinement procedure that discovers a per-VLM evaluation taxonomy. An initial taxonomy is constructed by prompting the target VLM to enumerate physics errors on a small set of videos and clustering the resulting descriptions. The taxonomy is then refined through a diagnostic step: we calibrate the VLM's per-dimension scores to human physical-commonsense ratings, diagnose which dimensions it scores unreliably or redundantly, and prompt an LLM to repair them, iterating until convergence. We further instantiate this procedure as a benchmark and apply it to 16 VLMs spanning eight model families. The refined taxonomy outperforms the global-schema baseline on held-out videos for every VLM tested, with a mean relative improvement of approximately 32%. Beyond aggregate accuracy, the per-VLM profiles expose model-specific blind spots that overall rankings cannot anticipate, with reliability patterns differing markedly across model families.
