Search papers, labs, and topics across Lattice.
This study evaluates the ability of multimodal large language models (MLLMs) to provide open-ended aesthetic critiques, contrasting their outputs with human critiques using the Reddit Photo Critique Dataset. The authors find that traditional reference-based similarity metrics fail to accurately capture the nuanced differences between MLLM critiques and human evaluations, revealing that MLLMs tend to produce more verbose and less selective critiques. Ultimately, the research highlights the limitations of current evaluation methods and suggests that they favor fluency over the specificity and depth characteristic of human critique.
MLLMs generate critiques that are more verbose and less selective than human reviewers, raising questions about the validity of traditional evaluation metrics.
Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single correct answer, and most aesthetic evaluation has measured models against numeric scores rather than the written critiques people actually give. We evaluate MLLM critiques against ranked human references and ask whether they are close to human ones. Using the Reddit Photo Critique Dataset, we score five open-weight MLLMs against multiple ranked human critiques per photo with reference-based similarity metrics, under six prompt conditions that disentangle persona framing, aspect hinting, length control, and single- versus multi-pass generation, and add an image-grounding control that feeds each model the wrong photograph. We find that reference-based similarity gives a misleading picture. Stricter lexical and learned metrics show only weak alignment with human critiques, while a coarse embedding cosine reports broad topical overlap that the grounding control traces to a stable house style rather than image-specific observation. Behaviorally, the models diverge from humans in consistent ways the scores do not surface: even under a length cap they write two to three times as much, cover nearly every aesthetic aspect where humans are selective, engage each aspect more uniformly and at greater depth, and repeat themselves across critiques of the same photo where humans vary. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique, and discuss implications for evaluating and training open-ended multimodal generation.
