Search papers, labs, and topics across Lattice.
This study introduces an inverted evaluation approach where LLM-powered robots assess their own interactions using standardized questionnaires, revealing a significant discrepancy in their ratings of comfort and strangeness compared to human evaluations. While the robots demonstrated moderate-to-strong agreement on engagement dimensions like satisfaction and enjoyment, they systematically inverted the comfort/strangeness dimension, indicating a fundamental misunderstanding of these constructs. The findings suggest that LLMs currently lack the necessary internal affective states to accurately evaluate certain aspects of human-robot interaction, highlighting the need for additional modalities to enhance evaluation accuracy.
LLM-powered robots misinterpret comfort and strangeness in human-robot interactions, conflating engagement with comfort in their self-assessments.
Human-robot interaction (HRI) evaluation relies almost exclusively on human-completed questionnaires, leaving the robot's perspective unexamined. We propose an \textit{inverted evaluation}, in which LLM-powered robots complete the same standardized instruments from their own perspective, and test whether these ratings agree with human ground truth. In Study~1, five LLMs completed HRI-CUES, Godspeed, and RoSAS questionnaires for 25~interactions ($N = 1{,}522$ evaluations) from the HRI-CUES dataset. LLMs achieved moderate-to-strong agreement on engagement dimensions (satisfaction $r$ up to $.65$ and enjoyment $r$ up to $.72$) with excellent test-retest reliability (ICC $\geq .82$), but \textit{systematically inverted} the comfort/strangeness dimension ($r = -.44$ to $-.67$, all $p < .05$), conflating engagement with comfort. In Study~2, a Nao robot running Claude~Sonnet~4.5 replicated these patterns in live interactions ($N = 4$), including real-time turn-by-turn assessment. The strangeness failure persisted across five models, synthetic controls, and embodied deployment for two participants. We argue that current LLM-based robots lack access to the internal affective states needed to assess constructs like strangeness, and that inverted evaluation requires supplementary modalities (e.g., physiological signals, gaze, proxemics) to move beyond behavioral proxies. These findings establish boundary conditions for using LLMs as interaction evaluators in HRI.
